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1.  INTRODUCTION 


The  purpose  of  this  guidebook  is  to  provide  Air  Force  Program 
Office  engineering  and  management  personnel  with  information  that  will 
help  them  plan,  prepare  for,  and  conduct  technical  reviews  and  audits  in 
connection  with  the  acquisition  of  Computer  Program  Configuration  Items 
(CPCI's)  for  airborne  systems. 

The  guidelines,  checklists,  and  references  in  this  guidebook  serve 
to  supplement  the  formal  requirements  in  MIL-STD-1521A  (USAF), 
"Technical  Reviews  and  Audits  for  Systems,  Equipment,  and  Computer 
Programs,  " and  are  directed  specifically  to  reviews  and  audits  for  com- 
puter programs. 

Figure  1-1  shows  an  idealized  life  cycle  model  of  an  airborne  system 
and  indicates  the  relationship  of  reviews  and  audits  to  the  system  life 
cycle.  Table  1-1  lists  the  major  types  of  reviews  and  audits  and  depicts 
their  relationship  to  steps  in  the  CPCI  acquisition  process.  A more 
detailed  picture  of  the  software  life  cycle.  Figure  1-2,  also  indicates  the 
relationships  of  reviews  and  audits  to  the  life  cycle  phases  and  to  an 
example  documentation  set.  Table  1-2  presents  in  greater  detail  the 
formal  reviews  and  audits  listed  in  Table  1-1,  with  summary  information 
on  the  purpose,  prerequisites,  and  items  to  be  reviewed  or  audited. 

1.1  PURPOSE  OF  REVIEWS  AND  AUDITS 

As  Tables  1-1  and  1-2  indicate,  the  purpose  of  technical  reviews 
and  audits  is  to  monitor  the  orderly  evolution  of  the  CPCI's  through  the 
sequential  steps  in  the  development  process  by  means  of  positive  definition 
and  control  of  documents  and  code  (performance  requirements,  interface 
requirements,  test  requirements,  test  definition,  design  requirements, 
design  solution,  code  and  test  results)  reviewed  at  benchmark  points  and 
corresponding  baselines  during  the  process.  The  major  features  of 
reviews  and  audits  are  summarized  in  Table  1-2,  which  also  serves  as  a 
convenient  checklist  and  guide.  More  detail  on  the  conduct  of  the  reviews 
and  audits  is  given  in  Section  4 of  this  guidebook. 


Review/Audit  * 


System  Requirements  Review  (SRR) 


System  Design  Review  (SDR) 


Preliminary  Design  Review  (PDR) 


Critical  Design  Review  (CDR)* 


Test  Readiness  Review  (TRR) 
(Informal  Review) 


Physical  Configuration  Audit  (PCAI 


Table  1 -1 , 


Formal  Baseline  * % 
Established 


Intermediate  Milestones 


FUNCTIONAL  BASELINE 


The  system  / subsystem  "functional 
baseline"  is  established  by  an 
acceptable  SRR  and  formal  cus- 
tomer approval  (authentication) 
of  the  System/System  Segment 
Spec  ification. 


ALLOCATED  BASELINE 


The  "allocated  baseline"  for 
a CPCI  is  established  by  an 
acceptable  SDR  and  formal 
customer  \pproval  (authen- 
tication) of  the  CPCI  Part  I 
Development  Specification. 


Preliminary  Design 


Detailed  Design 


CPCI  Code  Under  Internal 
Configuration  Control 


Functional  Configuration  Audit  (FCA) 
Formal  Qualification  Review  (FQR) 


Qualified  CPCI 


PRODUCT  BASELINE 


The  "product  baseline"  for  a 
CPCI  is  established  by  an  accept- 
able PCA  and  formal  customer 
approval  (authentication)  of  the 
CPCI  Pa  rt  II  Product  Specification. 


• Adeqi 
requl 
requi 


• Procg 
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Adeqti 

requi* 

requi*) 


• Procei 
design 


• Adequl 
requirj 
appio^ 


• Adequa 
requirj 
design.. 


• Proceg 


• Satisfa 
develoi 


Proceil 

testing: 


• CPCI  <| 
test  m< 
Spec  if  it 

• Procee 


• "As  bul 
its  tecD 
configu 


CPCI  A 


"The  purpose,  prerequisites,  end  items  to  be  reviewed  are  summarized  in  Table  1-2  and  discussed  in  detail  in  Section  4. 

The  two  elements  that  distinguish  "Forme)  Baselines"  from  "Intermediate  Milestones"  are  that 

(1)  formal  baselines  involve  formal  customer  approval  of  configuration  documentation  (specifications),  and 

(2)  approved  specifications  are  put  under  formal  configuration  control. 

For  complex  CPCI's,  ir  if  top-down  development  is  followed,  a series  of  "incremental"  CDR's  should  be  held. 

Test  Readiness  Review  (TRR):  Internal  review  by  CPCI  development  contractor  (customer  attendance  optional)  to  establil 
preparedness  (adequate  test  procedures,  availability  of  tools,  facilities  and  personnel,  adequate  CPCI  configuration  contf 
etc.)  to  commence  qualification/acceptance  testing. 


/ 


nts  Review  (SRR) 


Table  1-1.  Overview  of  the  Relationship 
of  Reviews/Ardits  to  Steps  in 
the  CPCI  Acquisition  Process 


Formal  Baseline  # * 
Established 


FUNCTIONAL  BASELINE 

The  system /subsystem  "functional 
baseline"  is  established  by  an 
acceptable  SRR  and  formal  cus- 
tomer approval  (authentication) 
of  the  System/System  Segment 
Specification. 


ALLOCATED  BASELINE 

The  "allocated  baseline"  for 
a CPCI  is  established  by  an 
acceptable  SDR  and  formal 
customer  approval  (authen- 
tication) of  the  CPCI  Part  I 
Development  Specification. 


Intermediate  Milestones 


juration  Audit  (FCA) 
ition  Review  (FQR) 


I 


ration  Audit  (PCAl 


Preliminary  Design 


Detailed  Design 


CPCI  Code  Under  Internal 
Configuration  Control 


PRODUCT  BASELINE 

The  "product  baseline"  for  a 
CPCI  is  established  by  an  accept- 
able PC  A and  formal  customer 
approval  (authentication)  of  the 
CPCI  Part  II  Product  Specification. 


Comment 


• Adequate  allocation  of  mission 
requirements  to  system 
requirements 

• Proceed  with  requirements 
al  ocation  to  Cl's  and  CPCI's 


• Adequate  allocation  of  system 
requirements  to  CPCI 
requirements 

• Proceed  with  CPCI  preliminary 
design  (design  ana'ysis) 


• Adequate  allocation  of  CPCI 
requirements  to  a basic  design 
approach 

• Proceed  to  detailed  design 


• Adequate  allocation  of  CPCI 
requirements  to  a detailed 
design 

• Proceed  to  code  and  test 


• Satisfactory  completion  of 
development  testing 

• Proceed  with  qualification 
testing 


• CPCI  qualification/acceptance 
test  meets  Development 
Specification  requirements 

• Proceed  with  PCA 


• "As  built"  CPCI  compatible  with 
its  technical  documentation  and 
configuration  management  records 

• CPCI  Acceptance/Delivery 


rercquisitcs.  and  items  to  be  reviewed  are  summarized  in  Table  1-2  and  discussed  in  detail  in  Section  4. 

its  that  distinguish  "Formal  Baselines"  from  "Intermediate  Milestones"  are  that 

ilines  involve  formal  customer  approval  of  configuration  documentation  (specifications),  and 

pecificat ions  are  put  under  formal  configuration  control. 

PCI's,  or  if  top-down  development  is  followed,  a series  of  "incremental  ' CDR's  should  be  held. 

Review  (TRR):  Internal  review  by  CPCI  development  contractor  (customer  attendance  optional)  to  establish 
tdequate  test  procedures,  availability  of  tools,  facilities  and  personnel,  adequate  CPCI  configuration  control, 
nee  qualification/acceptance  testing. 
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Table  1-2.  Overvl 


Audita 

(largd 

pouch] 


mm  Raelaw/ Audit  h Cewdama 

Formal 

■amhtte 

istebllsliid 

Purpoao 

"--I 

srr 

(System  Requirement*  Review) 

• Inprocest  reviews  ueuelfy  conducted  at  the  end 
of  the  System  Conceptual  Phase  and  early  m the 
System  Validation  Phase  if  the  Conceptual 

Phase  effort  was  performed  in-home  by  the 
government. 

Functional  Baseline 

• Establish  System /System  Segment  requirements  (System  Functional 
Baseline)  as  a basis  for  the  allocation  of  requirements  to  hardware 
Configuration  items  (Cl's)  and  Computer  Program  Configuration 

Items  (CPCI’s)  in  the  System  Validation  Phase. 

• Evaluate  the  total  systems  engineering  activity  for  responsimnoM  to 
the  Statement  of  Wort  end  the  System /System  Segment  requirements. 

• Provide  Procuring  Activity  direction  as  necessary,  to  the  contractor 
for  continuing  the  technical  program  and  system  optimisation 

• System/System  Segment  Specification 

• Statement  of  Work 

• Totality  of 

aUocetic!i> 

SOR 

(System  Design  Review) 

• Final  review  prior  to  submittal  of  the  Validation 
Phase  products  or  as  the  initial  review  in  the  Full 
Scale  Development  Phase  for  lystems/subsysteim 
not  requiring  • formal  Validation  Phase 

• Availability  of  draft  Pert  | Development  Specif  i 
cations,  for  all  CPCI's.  is  a prerequisite  'or  this 
review. 

Allocated  Baseline 

• E valuate  the  optimisation,  traceability,  correlation,  completeness  end 
risk  of  the  allocated  functional  requirements  (Allocated  Baseline), 
including  the  corresponding  test  requirements,  in  fulfilling  the  System/ 
System  Segment  Specification  requirements. 

• Ensure  that  the  updated  System /System  Segment  Specification  it 
adequate  end  cost  effective  in  satisfying  validated  mission  requirement. 

• Ensure  that  a technical  understanding  of  requirements  end  supporting 
systems  engineering  results  has  been  attained,  end  Procuring  Activity 
technical  direction,  as  necessary,  provided  to  the  contractor. 

• Evaluate  the  contractors  concept  for  development/support  of  the 

CPCI’s  for  completeness,  adequacy,  and  schedule/cost  realism. 

• Upon  approval  of  each  CPCI  Pert  I Development  Specification,  satis- 
faction of  this  review  establishes  the  Allocated  Baseline  for  individual 
CPCI’s  and  provides  the  basis  to  proceed  with  prelimuiery  design  of 
the  CPCI's. 

• Updated  System/ System  figment  Specification 

• Preliminary  Part  I Development  Specification  for 
each  CPCI 

• Computer  Program  Development  Plan  (CPOP) 

• Computer  Program  Configuration  Management  Plan 
(may  be  a component  of  the  CPOP) 

• Addition* 
motion /set 
d location, 
bility,  rqk 
cutting.  M| 

• Summery  | 
allocation  1 

• Oescnpttoa 

POR 

(Preliminary  Design  Review) 

• The  POR  for  each  CPCI  (may  be  grouped)  is 
held  early  in  the  Full-Scale  Development  Phase 
(between  SDR  and  CDH)  when  sufficient  design 
analysis  hat  been  eccompiished  to  arrive  at  a 
computer  program  architecture  end  overall 
modular  structure  which  will  provide  the 
basis  for  detailed  design. 

• Availability  of  an  authenticated  Part  1 Develop 
mens  Specification  is  a prerequisite  for  any 

CPCI  to  bt  POR'd. 

N/A 

• Evaluate  the  basic  design  approach  for  completeness,  adequacy,  and 
compatibility  with  at  locked  requiramantt  (Part  1 Development 
Specification). 

• Review  ell  changes  to  the  System/System  Segment  Specification  and 

Part  1 Development  Specification  to  ensure  that  they  are  properly 
incorporated  in  the  basic  design  approach,  the  draft  Pert  II  Product 
Specification,  and  test  planning. 

• Review-  alt  detaile>  ’ functional  interfaces  and  corresponding  test 
requirements  with  hardware  Cl’s  end  other  C ’Cl's. 

• Review  the  CPCI  interactions  with  Human  Factor  requirements. 

Review  all  man-machine  interfaces  for  feasibility,  adequacy,  and 
completeness. 

• Review  test  planning  documentation  to  ensure  that  the  test  program 
satisfies  the  tost  requirements  specified  in  Section  4 of  the  System/ 
System  Segment  Specification  and  Section  4 of  the  Pert  1 Develop 
ment  Specification. 

• Review  status  of  all  negative  end  provisional  entries  such  as  "not 
applicable"  (N/A)  or  "to  be  determined”  (TBO)  m Section  4 of  the 
System/System  Segment  Specification  and  Pert  1 Development 
Specification.  Review  ell  positive  entries  for  technical  adequacy. 

• Updated  System/System  Segment  Specification 
(if  necessary) 

• Final  Pert  1 Development  Specification  for  each  CPCI. 

• Partial  Pert  1 1 Prodect  Specification  for  each  CPCI . 

• Preliminary  Qualification  (Acceptance)  Test  Plan  for 
each  CPCI. 

• Preliminary  Dele  Bam  Document  for  each  CPCI. 

I 

COR 

(Critical  Oetifn  Review) 

• Conducted  between  PDR  and  FCA  when 
detailed  oetign  it  complete  for  the  CPCI 
components  (appropriate  subset  if  incremental 
COR’t  ere  scheduled) 

• Draft  Pert  II  Product  Specification  (deluding 
code  listmp)  it  available  if  single  CDR  for 
total  CPCI 

• Appropriate  portions  of  Pert  II  Product  Speci- 
fication is  available  if  review  it  scheduled  as  an 
incremental  CDR.  In  this  case  complete  draft 

Pbrt  II  Product  Specification  (excluding  code 
listings)  available  at  final  CDR. 

N/A 

• Review/evaluation  of  the  detailed  design  of  the  CPCI  (selected  com 
porents  if  incremental  CDR)  for  adequacy  and  completeness. 

• Formal  identification  of  specific  computer  programming  ("build  to") 
documentation  which  will  be  released  for  coding  and  testing. 

• Approval  of  Qualification  (acceptance)  Test  Plan  provides  ■ controlled 
definition  of  the  protect's  acceptance  test  program. 

• Review/evaluate  plant  for  supporting  end  maintaining  the  CPCI  includ- 
ing ell  necessary  hardware,  support  software,  and  documentation. 

• Pert  1 Development  Specification  indudmg  ell 
approved  TCP's 

• Preliminary  Pert  II  Product  Specification  (excluding 
code  listings).  If  an  mere  mental  COR.  only  compo 
nents  under  review  mould  be  induded. 

• Final  Qualification  (Acceptance)  Test  Plan 

• Updated  Date  Bast  Document 

• Preliminary  Computer  Programming  Manuel 

• Preliminary  Operator’*  (Dear’s)  Manuel 

• POR  Minutes 

• Updated* 
analyte*  | 

scientific^ 

• Applicable 

(TCP'S)  to 

• DeignN 

• Status  of  • 

• Planning* 

• Dane  loot— 

fZA 

(Function el  Configuration  Audit) 

• Conducted  after  CDR  end  prior  to  PCA  when 
sufficient  quelification/ecoeptence  testing  has 
been  performed  to  verify  all  of  the  Part  1 
Development  Specification  requirements 

• May  be  conducted  on  a progressive  (incremental) 
bests  il  to  designated  by  the  Procuring  Activity. 

• For  situations  where  CPCI  qttehf  cation  can  only 
be  determined  throu^i  integrated  system*  test 
mg.  FCA’*  for  such  CPCI’s  will  not  be  considered 
complete  until  completion, 'audit  of  sud*  testing. 
This  normally  requires  a separate  FOR  pott-PCA 

• Authentication  by  the  Procuring  Activity  of  the 
Functional  end/or  Al'oceted  Baselines  it  a 
prerequisite  to  FCA. 

N/A 

• Verify  that  the  CPCI  actual  (test)  performance  totally  complies  with 
the  Pert  1 Development  Specification  requirements. 

• Determine  adequacy  of  analysis  or  simulation  results  where  perform- 
ance parameters  cannot  completely  be  verified  by  test. 

• L .a  uete  the  contractor’s  proposed  solutions  for  any  requirements 
stated  m the  Pert  1 Development  Specification  that  could  not  be  met. 

• t men  ine  end  evaluate  tU  approved  TCP's  to  ensure  that  they  were 
incorporated  and  verified  during  ruaiifical ion/acceptance  testing. 

• F * emme  the  Quaiificetion/Acceptan  » Test  Report  to  verify  it  is  an 
accurate  end  complete  description  of  the  auelifieetion/ecceptenca 
testing. 

• E Remine  POR  end  CDR  minutes  to  assure  that  ell  findings  have  fcsen 
incorporated  end  completed. 

• Authenticated  System/System  Segment  Spedficetion 
•nd  Part  1 Development  Spedficetion. 

• Oreft  CPCI  Part  ll  Product  Spedficetion  (complete) 

• CPCI  Test  Plant.  Tam  Procedures,  end  Test  Resutts 

• PDR  end  CDR  mmetes. 

1 

PCA 

(Physical  Configures  .on  Audit) 

• Conducted  between  PCA  and  FOR  (when  sepa- 
rate FDR  is  required)  when  ell  required  audit 
date  is  available  (nommeMy)  30  days  after  sub 
nvttel  of  the  draft  Fmel  Pert  II  Product 
specification  to  the  Procuring  Activity. 

Product  Baseline 

• Formal  * seminal  ion  of  coded  ("as  built")  version  of  the  CPCI  against 
its  techn.cal  documentation  and  of  the  configuration  men agament 
records  pertinent  to  the  CPCI  to  establish  the  CPCI  Product  Baseline. 

• Evaluate  all' CPCI  configuration  differences  between  FCA  and  PCA 
versions  to  ensure  CPCI  functional  characteristics  are  not  degraded. 

• Review  FCA  minutes  to  assure  that  all  findings  haw  been 
incorporated  end  completed 

• Audit  the  contractor’s  engineering  release  and  change  control  system 
to  ensure  that  it  is  adequate  to  properly  control  the  precasting  end 
formal  release  of  engineering  changes  sfter  CPCI  acceptance/delivery. 

• Satisfactory  completion  of  the  PCA  end  format  approval  by  the 

Procuring  Activity  of  the  CPCI  Pert  •)  PrpctaA  Specification  estab- 
lishes the  CPCI  Product  Baseline. 

• Accepted  CPCI’s  ere  delivered  in  accordance  with  contract 
i equipments. 

• Authenticated  CPCI  Port  1 Development  Spedficetion 

• Draft  Final  CPCI  Part  II  Product  Spedficetion. 

• CPCI  Test  Plans.  Tsst  Fre— dmes.  and  Test  Reports. 

• Oraft  Final  Operator’s  (User’s)  Manuel. 

• Version  Description  Document  (VDO). 

• FCA  Minutes. 

• Prepared  DO  Form  MO  or  equivalent. 

• List  Of  dl 

awaiting  I 

• List  deli— 

• List  of  aR 

• List  of  a* 

• CPCI  cold 

• CPCI  mad 

. Oaten— 

POR 

(Formal  Qualification  Review) 

• —hen  feawble.  the  FDR  w eon— mad  with  the 
FCA. 

• For  situations  m which  the  Fart  1 Development 
Specification  requirements  cannot  totally  bo 
sorifiod  by  the  testing  avelueiad  at  FCA  leg  . 

system  testing),  a teperets  FOR  end  be  can 
ducted  poet -FCA.  when  the  iseceseery  tests 
have  been  satisfactorily  —mpfiwd.  to  suable 

CPCI  certification. 

N/A 

• Seme  as  FCA.  but  for  FOR  separate  from  FCA.  fhe  foHowmgmust 

be  accomplished 

• Review  FCA  minuses  to  ensure  that  eM  findings  have  been  incor- 
porated end  completed;  the  FOR  is  considered  an  extension  of 
the  FCA. 

• Review  end  es-uase  additional  quellficetion/scceptencs  test  date, 
together  with  the  FCA  ffndtnp.  to  ensure  qualification  of  the 

CPCI  against  the  total  set  of  require menti  in  the  CPCI  Part  1 
OhdspPtq  Sped  licet  ion. 

• Seme  as  FCA.  plus 

• Additional  test  results 

• FCA  Mmutet 

• Serna  as? 

' 

* 

V 

■ 
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Table  1-2.  Overview;  Formal  Reviews  and 
Audits  for  Software  Acquisition 
(larger  version  of  this  table  in 
pouch  inside  back  cover) 

r 

Esesse/Audil  h Cenducsaa 

Nrajal 

Ptsrpeea 

f same  te  be  Reviewed  | 

MUM 

f—r, 1.- 

lapmsi  Rapa  ns.  ebj 

inn  uauolty  conquaeu  et  the  end 
l Concept  uei  Phase  end  eviy  m the 
Ihon  Phew  if  the  Concept ud 

Mi  performed  m-house  by  the 

Functional  Baseline 

• Establish  System /Syttem  Segment  requirements  (System  Functional 
Baaalme)  as  a basis  for  tha  allocation  of  requiramants  to  hardware 
Configuration  items  (Cl's)  and  Computer  Program  Configuration 

Items  (CPCI't)  in  tha  System  Validation  Phase. 

• Evaluate  the  total  systems  angmaaring  activity  for  responsiveness  to 
tha  Statement  of  Work  and  tha  System /System  Segment  requirements. 

• Provide  Procuring  Activity  direction,  at  necessary,  to  tha  contractor 
for  continuing  tha  technical  program  and  system  optimization 

• Svstem/Syttam  Sagment  Specification 

• Statement  of  Work 

• Totality  of  system  engineering  efforts  to  dale,  for  exempli,  mweien/syaaam 
requirements  analyses,  functional  Row  analyses,  preii  nursery  r squire  menu 
allocation,  life  cycle  cost  studies,  integration  and  teat  planning,  oenhguratten 
management  and  data  management  planning,  etc. 

Mr  «o  submittal  of  the  VMstion 
li  or  as  the  initial  raviaw  m tha  Full 
Mint  Phaea  for  syttems/siAsystaira 
• formal  Validation  Phase 

If  draft  Part  1 Development  Spaeth 

» CPCf's.  is  a prerequisite  fqr  dm 

Allocated  Baseline 

• Evaluate  the  optimization,  traceability . correlation,  completeness  and 
mk  of  the  allocated  functional  requirements  (Allocated  Baseline), 
including  the  corresponding  test  requirements,  in  fulfilling  the  System/ 
System  Segment  Specification  requirements 

• Ensure  that  the  updated  System /System  Segment  Specification  is 
adequate  and  ooet  effective  in  satisfying  validated  mission  requiramants. 

• Ensure  that  a technical  undemanding  of  requirements  and  supporting 
systems  engineering  results  has  been  attained,  end  Procuring  Activity 
technical  direction,  as  necessary,  provided  to  the  contractor. 

CPCI't  for  compfet toots,  adequacy,  and  tchedula/coet  realism 

• Upon  approval  of  ta<h  CPCI  Pert  1 Development  Specification,  satis- 
faction of  this  renew  establishes  the  Allocated  Baseline  for  individual 
CPCI't  and  provides  the  basis  to  proceed  with  preliminary  design  of 
the  CPCI't. 

• Updated  Svttem/Svstem  Segment  Specification 

• Preliminary  Part  1 Development  Specification  for 
each  CPCI 

• Computer  Program  Development  Plan  (CPOP) 

• Computer  Program  Configuration  Management  Plan 
(may  be  a component  of  tha  CPOP) 

mission /system  requirements  analyses,  fund  tonal  analyses,  requirements 
allocation,  system/ cost  effectiveness,  system  synthesis,  system  groevth  cape 
bility.  risk  analytes,  system  interlace  studies,  sitmg/timing  analytes,  life  cyda 
costing,  trade  studies,  etc. 

• Summary  presentation  of  the  mrstion/sysiem  requirements  and  requirement 
allocation  to  Cl's  and  CPCI’t. 

• Description  of  tha  CPCI  development/tupport  concept. 

m 

M/A 

• Evaluate  the  basic  design  approach  for  completeness,  adequacy,  and 
compatibility  with  allocated  requirements  (Part  1 Development 
Specification). 

• Review  all  changes  to  tha  Svstem/Syttam  Segment  Specification  and 

Part  1 Development  Specification  to  ensure  thet  they  ere  properly 
incorporated  m the  basic  design  approach,  the  draft  Part  II  Product 
Specification,  and  test  planning 

• Review  ell  detailed  functional  interfaces  and  corresponding  test 
requirements  with  hardware  Cl’s  end  other  CPCI't. 

• Review  the  CPCI  interact  tom  with  Human  Factor  requirements. 

Review  all  man-machine  interfaces  for  feasibility,  adequacy,  and 
completeness 

• Review  test  planning  documantttion  to  ensure  that  the  test  program 
satisfies  the  test  requirements  specified  in  Section  4 of  the  System/ 

Syttem  Segment  Specification  and  Section  4 of  the  Part  1 Develop 
ment  Specification. 

• Review  status  of  all  negative  and  provisional  entries  such  as  "not 
applicable"  (M/A)  or  "to  be  determined"  (TBO)  in  Section  4 of  the 
Syttem/System  Segment  Specification  and  Pert  1 Development 
Specification.  Review  all  positive  entrws  for  technical  adequacy 

• Updated  Syttem/System  Segment  Specification 
(if  necessary) 

• Final  Pan  t Development  Specification  for  aach  CPCI 

• °artial  Part  II  Product  Specification  for  aach  CPCI. 

• Preliminary  Qualification  (Acceptance)  Test  Plan  for 
each  CPCI 

• Preliminary  Data  Base  Document  for  each  CPCI. 

• Available  systems  eigineenng/detign  data,  for  example : 

• CPCI  functional  flows 

• CPCI  storage  allocation  cherts. 

• Sizing  and  timing  astimatat  and  budgets 

• CPCI  control  functions,  mdudmg  executive  control  end  stert/recotmry  features 

• Overall  hierarchical  structure  for  each  CPCI  and  accompanying  rationale 

• Data  bate  structure  and  organization 

• Significant  design  tradeoff  studies 

• Interface  definition  between  the  CPCI.  hanhvart  Cl's  end  other  CPCI't 

• Identification  of  unique  security  requirements,  if  any,  end  design  approach 
to  satisfy  tham 

• Identification  of  any  reantrancy  requirements  and  approach  for  imp*# mentation 
and  test. 

• Description  of  the  availability,  adequacy,  and  planned  utilization  of  toots  and 
facilities  for  CPCI  development  including  Syttem/CPCI  exercising. 

• Description  of  any  special  tooh.  non  deliver  able  under  tha  contract,  but  which 
ara  planned  to  be  used  during  CPCI  development. 

• Status  of  all  Sep's  and  DPP's  again** ;!«  CPCI . 

•ween  POR  and  FCA  when 

i is  complete  for  the  CPCI 
Ippropriete  subset  if  incremental 

edwtad). 

Iboduct  Specification  (excluding 
•m  available  if  smgia  CDR  for 

lemon*  of  Pan  II  Product  Spso 
Nable  if  review  is  scheduled  as  an 

SDR  In  this  cesa  complete  draft 

It  Specification  (excluding  coda 
Meet  final  CDR 

N/A 

• Review /evaluation  of  the  detailed  design  of  the  CPCI  (selected  com- 
ponentt  if  incremental  CDR)  for  adequacy  and  completeness. 

• Formal  identification  of  specific  computer  programming  ( "build  to") 
documentation  which  will  be  released  for  coding  and  testing. 

• Approval  of  Qualification  (acceptance)  Test  Plan  provides  a controlled 
definition  of  the  protect’s  acceptance  test  program. 

• Revww/evaluete  plant  for  supporting  and  maintaining  tha  CPCI  includ- 
mg  all  necessary  hardware,  support  software,  and  documentation. 

• Part  I Development  Specification  including  all 
approved  ECP*s. 

• Preliminary  Part  II  Product  Specification  (excluding 
coda  listings).  If  an  incremental  COR,  only  compo 
nentt  under  review  would  be  included. 

• Final  Qualification  (Acceptance)  Test  Plan 

• Updated  Data  Base  Document 

• Preliminary  Computer  Programming  Manual 

• Preliminary  Operator's  (User's)  Manual 

• POR  Minutes 

• Updated  systems  engineering  or  design  enelyses/studwt.  eg.,  sizing/timing 
analyses,  performance  studies,  accuracy  analyses,  algorithm  tradeoffs, 
scientific/environment  simulation  results,  etc. 

• Applicable  configuration  management  records  related  to  appromd  changes 
(ECP*S)  to  the  Part  1 Development  Specification 

• Design  Problem  Reports  (DPR's). 

• Status  of  all  ECP"s  and  DPR's  (approved,  outstanding,  'erected I . 

• Planning  data  for  supporting  (maintaining!  CPCI  post  deployment. 

• Development  Test  Plan. 

N/A 

• Verify  thet  the  CPCI  actual  (test)  performance  totally  complies  with 
the  Part  1 Development  Specification  requirements 

• Determine  adequacy  of  analysts  o-  simulation  results  where  perform- 
ance parameters  cannot  completely  be  verified  by  test. 

• E valuete  the  contractor's  proposed  solutions  for  any  requirements 
stated  m the  Pert  1 Development  Specification  that  could  not  be  mat 

• Examine  and  evaluate  all  approved  ECP*s  to  ensure  that  they  were 
incorporated  and  verified  during  quelificetion/acoeptance  tatting 

• Fxamine  tha  Qualification/ Acceptance  Test  Report  to  verify  it  is  an 
accurate  and  complete  description  of  the  queiification/acceptance 
testing 

• E xemme  POR  and  CDR  minutes  *o  assure  that  all  findings  have  bean 
incorporated  and  completed 

• Authenticated  Syttvm/System  Segment  Specification 
and  Pert  1 Development  Specification. 

• Oraft  CPCI  Pait  II  Product  Specification  (compfate) 

• CPC)  Test  Plans.  Test  Procedures,  and  Test  Results. 

• PDR  and  CDR  minutes 

• CPCI  configuration  management  status  records. 

• Briefings  to  the  FCA  teem  delineating  tha  test  results  and  findings  for  each  CPCI. 

• identification  of  any  performance  parameters  that  cermet  be  cpmptetefy  veofwd 
during  testing  and  demonstrated  adequacy  of  analysis  or  simulation  results  to 
satisfy  thasj  requirements. 

• A complete  list  of  functional  tests  required  by  the  Per*  1 Development  Specification 
but  not  yet  performed  (eg.,  to  be  performed  durmg  integrated  system  testing) 

• Identification  of  requirements  of  the  Pert  1 Development  Specification  that  the 
contractor  wet  not  able  to  satisfy,  if  any.  end  • proposed  solution  for  each  item 

Formal  • lamination  of  oodad  ("a*  built")  version  of  the  CPC  I against 
hi  technical  documentation  and  of  tha  configuration  management 
racordt  partmant  to  the  CPC  I to  establish  tha  CPC  I Product  Baseline. 
Evaluate  aN  CPC  I configuration  differences  bataraan  FCA  and  PCA 
versions  to  ensure  CPC  I functional  eheracter  isties  ara  not  dagradad. 
Unaa  FCA  minutes  to  assure  ttiat  all  findings  hem  baan 
incorporated  and  completed 

Audit  tha  contractor'!  engineering  rafaata  and  changa  control  system 
to  amura  that  it  it  adequate  to  property  control  tha  processing  and 
formal  release  of  engineering  changes  after  CPC  I eceeptence/deliwry 
Satisfactory  completion  of  the  PCA  and  formal  approael  by  tha 
Procuring  Activity  of  tha  CPCI  Port  II  Product  Specification  estab 
Mhos  the  CPCI  Product  Baseline. 


Authenticated  CPC'  Part  I Development  Specification 
Draft  Final  CPCI  Pan  II  Product  Specification 
CPCI  Test  Plant,  Test  Procedures,  and  Test  Reports. 
Draft  Final  Operator's  (User's)  Manual 
Oraft  Final  Computer  Programming  Manual. 

Version  Description  Document  (VDD). 

FCA  Minutes 

Prepated  DD  Form  250  or  equivalent. 


• List  of  ail  devienona/weivert  against  the  CPCI.  either  approved,  or  outstanding 
awaitmg  approval  by  the  Procuring  Activity 

• List  delineating  both  approved  and  outstanding  changes  ((CP's)  against  tha  CPCI 

• List  of  all  required  changes  not  yet  completed 

• List  of  all  changes  actually  mad#  during  test. 

• CPCI  configuration  management  status  accounting  records. 

• CPCI  master  tape  end  current  listing  end  ftovechertt. 

• Description  of  ontrector's  engineering  relee h end  configure! ion  control  systems. 


• FCA  Minutes 


Structured  reviews  and  audits  allow  the  government  to 

e ascertain  that  the  contractor  is  developing  the  CPCJ's  in 
an  orderly  process  based  on  well-thought-out  and 
clearly  defined  sequential  steps; 

• determine  that  each  step  is  complete  before  the  next 
development  step  is  started; 

a ensure  that  the  contractor  has  not  misinterpreted  or 
misunderstood  the  government's  requirements; 

• ensure  that  the  contractor  is  not  neglecting  some  facet 
of  software  development  such  as  support  software;  and 

• identify  major  problem  areas  without  attempting  to 
determine  the  absolute  technical  correctness  of  each 
document  from  the  contractor  (in  particular,  the 
government  contractually  approves  requirements, 
acceptance  test  plan,  and  final  configuration  but  should 
not  formally  "approve"  the  design). 

From  the  contractor's  point  of  view,  effective  reviews  and  audits 
allow  him  to: 

e determine  periodically  that  all  agencies  involved  agree 
on  the  correctness  of  his  approach, 

e obtain  appropriate  technical  direction  early  enough  to 
avoid  the  need  for  expensive  rework  of  downstream 
products, 

e obtain  formal  approval  of  requirements,  acceptance  test 
plans,  and  adequacy  of  qualification  testing  (but  not  of 
design),  and 

e close  out  the  project  by  obtaining  formal  approval  and 
acceptance  of  the  final  product  and  its  accompanying 
documentation. 

A review  or  audit  should  be  viewed  as  a mini-project,  subject  to 
all  the  rules  that  apply  to  any  project.  The  first  of  these  is  that  a plan 
must  be  prepared.  The  government  program  manager  or  project  engineer 
must  identify  the  personnel  to  be  involved  and  prepare  a plan  for  each 
review  or  audit;  he  should  also  make  sure  that  his  contractor  counter- 
part does  the  same.  A detailed  agenda  for  the  review  or  audit  meeting 
should  be  prepared  by  the  contractor  and  coordinated  with  the  govern- 
ment 3 to  4 weeks  before  Ihe  review. 
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The  overall  definition  of  reviews  and  audits,  indicating  how  many 
there  should  be.  nominal  schedules,  and  the  content  and  scope  of  each, 
should  be  included  in  the  governing  contractual  documentation  (contract 
clauses,  SOW.  CDRL). 

The  government  representative  in  charge  of  supporting  a review  or 
audit  should  be  aware  that  reviews  and  audits  can  serve  purposes  in  addi- 
tion to  those  explicitly  stipulated  in  MIL-STD-1521A  (USAF)  and  should 
consider  these  when  selecting  attendees  and  planning  his  support  for  the 
review  or  audit.  For  example,  it  is  important  to  assure:  (1)  system 
familiarisation  and  timely  exchange  of  technical  data  between  government 
agencies.  (2)  using  command  (TAC,  SAC)  participation  in  decisions 
involving  mission  operation  and  effectiveness,  (3)  using  command 
(AFLC/ALD  and  appropriate  Air  Logistics  Center)  involvement  in  support 
concepts  and  unique  support  requirements,  etc. 

1.2  CONTENTS  OF  THE  GUIDEBOOK 

1.2.1  Section  1:  Introduction  provides  background  information,  in 
particular  the  relationship  of  reviews  and  audits  to  the  system  life  cycle 
and  the  CPCI  development  process.  This  section  also  contains  a summary 
of  the  purposes  of  reviews  and  audits. 

1.2.2  Section  2:  Relevant  Documents  references  the  government 
regulations,  specifications,  and  standards  relevant  to  formal  reviews 
and  audits. 

1.2.3  Section  3:  General  Guidelines  for  Reviews  and  Audits  provides 
general  guidance  to  the  Air  Force  Program  Office  and  engineering  per- 
sonnel in  the  preparation  and  conduct  of  technical  reviews  and  audits 
relating  to  the  acquisition  of  CPCI's  for  airborne  systems.  It  identifies 
the  responsibilities  of  the  participating  organizations  and  describes 
useful  techniques  for  planning  and  conducting  reviews  and  audits. 
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1.2.4  Section  4:  Guidance  for  Individual  Reviews  and  Audits  provides 
detailed  guidance  for  each  review  and  audit  in  the  development  cycle.  In 
each  case,  the  purpose  and  objective  of  the  review  or  audit  are  defined, 
the  documentation  and  technical /management  data  to  be  evaluated  are 
listed,  specific  requirements  are  given  and  post-review  activities  are 
described. 

1.2.5  Appendix:  Bibliography  of  Government  Documents  provides 
additional  sources  of  background  information  related  to  reviews  and 
audits. 


2.  RELEVANT  DOCUMENTS 


The  following  documents  are  important  sources  of  information 
relevant  to  formal  reviews  and  audits.  Additional  sources  of  background 
information  related  to  reviews  and  audits  are  provided  in  the  Appendix. 


AFR  800-14  Volume  I 

Acquisition  Management; 
Management  of  Computer  Resources 
in  Systems 

AFR  800-14  Volume  II 

Acquisition  Management; 

Acquisition  and  Support  Procedures 
for  Computer  Resources  in  Systems 

MIL  -STD -1 521 A (USAF) 

Technical  Reviews  and  Audits  for 
Systems,  Equipment,  and  Computer 
Programs 

MIL-STD-490 

Specification  Standards 

MIL -STD -499 A 
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Engineering  Management 

MIL -STD-483  (USAF) 

Configuration  Management  Practices 
'for  Systems,  Equipment,  Munitions, 

* and  Computer  Programs 

MIL -STD-480 

Configuration  Control  - Engineering 
Changes,  Deviations  and  Waivers 

MIL  -STD -481 A 

Configuration  Control  - Engineering 
Changes,  Deviations,  and  Waivers 
(Short  Form) 

MIL -S- 52779  (AD) 

Software  Quality  Assurance  Program 
Requirements 

3.  GENERAL  GUIDELINES  FOR  REVIEWS  AND  AUDITS 


The  agencies  having  direct  responsibilities  for  reviews  and  audits 


the  procuring  agency  (with  any  direct  support  organizations 
such  as  Aerospace  Corporation,  Mitre  Corporation,  SETA 
contractors,  etc.  ), 


• subcontractors,  vendors,  suppliers  (if  software  is  being 
developed  under  subcontract  to  prime  contractor). 

Otjier  organizations  that  are  important  in  avionics  procurement 
and  that  may  be  appropriately  involved  in  reviews  and  audits  are 


• appropriate  ACO  and  AFPRO, 

• the  supporting  agency  (AFLC) 


* the  using  agency  (T AC,  SAC,  MAC,  ADCOM) 


• system  external  interfaces  (EW,  ATE,  C&C,  etc.). 

Reviews  and  audits  are  co -chaired  by  the  procuring  agency  and  the  con- 
tractor. However,  the  government  project  officer  is  responsible  for 

• defining  goals  and  ensuring  that  they  are  met, 

• keeping  affected  government  organizations  involved 
(e.g.  , providing  for  supporter  and  user  representation), 

• ensuring  that  the  review  is  carried  out  to  the  appropriate 
depth  (e.  g.  , providing  appropriate  technical  support  for 
the  review). 

Table  3-1  summarizes  the  responsibilities  of  both  the  contractor  and  the 
procuring  agency. 


Table  3-1.  Responsibilities  (Reviews /Audits) 


Procuring  Activity 

Contractor(s) 

e Provide  Co -Chairperson 

* Establish  review/audit  team, 
schedule  review  of  advance 
material  and  provide  coordi- 
nated comments  to  the  con- 
tractor in  advance  of  the 
review/audit  meeting;  assure 
continuity  in  the  review  team 
where  possible 

e Provide  name,  organization, 
and  security  clearance  of 
each  member  of  review /audit 
team 

e Define  goal*  and  ensure  that 

they  are  met 

* Keep  affected  government 
organizations  involved  (e.  g.  , 
provide  for  user  and  sup- 
porter representation) 

* Ensure  that  the  review/audit 
is  carried  out  to  the  appropri- 
ate depth  (e.  g. , provide 
appropriate  technical  support 
for  the  review  /audit) 

* Review  daily  minutes  and 
ensure  that  they  accurately 
reflect  all  significant  pro- 
curing activity  inputs 

s Co-sign 3 official  meeting 
minutes 

* Provide  technical  direction, 
as  necessary  (within  scope  of 
contractual  requirements) 

* Provide  formal  (i.e.,  written) 
acknowledgement  to  contractor 
of  review/audit  accomplish- 
ment after  receiving  official 
minutes 

• Review /audit  approval 

• Review/eudit  contingent 
approval 

• Review/audit  disapproval 

* Perform  follow-up  on  all  action 
items  resulting  from  review/ 
audit 

• Provide  Co-Chairperson 

• Establish  agenda,  time,  place 

• Furnish  meeting  facilities 
(conference  rooms,  etc.  ) 

• In  advance  of  the  review /audit, 
provide  appropriate  documen- 
tation and  supporting  engineering 
data  (i.e.  , items  to  be 
reviewed) 

• Ensure  that  review/audit 
schedule  is  compatible  with 
availability  of  contract  articles 
and  necessary  supporting  data 

• Prepare  and  present  expository 
briefings  as  appropriate 

• Provide  stenograpner  or  other 
acceptable  method  to  record 
inputs  to  meeting  minutes 

• Perform  daily  compilation, 
and  coordination  with  customer, 
of  meeting  minutes 

• Co -sign  * and  publish  a.*.d 
distribute  official  meeting 
minutes 

• Perform  follow-up  on  all  action 
items  resulting  from  review/ 
audit 

MIL-STD-1521A(USAF)  doe*  not  explicitly  require  that  the  official  minute* 
be  co-signed;  however,  it  i*  recommended  a*  a prudent  practice  and  evidence 
of  mutual  agreement*,  acceptance  of  action  items,  etc. 
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for  the  whole  review  cycle  (plan,  read,  listen,  comment,  follow-up); 
obtain  commitments  from  your  management  or  from  the  organizations 
supplying  the  reviewers  (see  Section  3.3.2  for  guidelines). 


i 


Significant  "homework"  is  recommended  prior  to  receiving  the 
explicit  advance  data  package  associated  with  a specific  review  and/or 
audit.  For  example,  review  or  audit  team  members  should  familiarize 
themselves  with  such  items  as  the  ROC,  PMO,  PMP,  CPDP,  CRISP, 
RFP,  contractor  proposals,  resulting  contracts  (particularly  the  SOW, 

♦i 

CDRL  and  specifications/exhibits,  if  any),  and  the  official  minutes  and 
data  packages  from  any  prior  reviews  and  audits. 

Formal  reviews  and  audits  are  tied  to  major  milestones  on  the 

project  and  to  the  related  CDRL  deliverables.  Your  superiors  will 

*v  * • 

expect  a report  on  the  review,  and  you  should  learn  in  advance  what 
areas  are  of  primary  concern  and  place  corresponding  requirements 
on  your  contractor.  The  applicable  parts  of  MIL -STD-1521A  are  dis- 
cussed in  Section  4 for  each  review  and  audit. 


As  part  of  preparing  the  review  plan,  you  will  need  to  evaluate  the 
review /audit  and  determine  the  following: 

• goals  of  the  review  or  audit  in  specific  terras  (i.e.,  not 
goals  that  could  apply  to  any  review); 


• skills  needed  on  your  team; 

0 decisions  you  must  make  (e.g.,  authorization  to  proceed  to 
detailed  design)  and  what  the  review  must  show  for  you 
to  be  able  to  make  those  decisions;  and 


• who  might  be  affected  by  the  results  of  the  review  or 

audit  (this  will  suggest  persons  who  should  be  invited  for 
information  or  concurrence). 

Recognize  that  a good  review/audit  costs  money;  include  the  cost 
in  your  project  budget.  Be  sure  your  contractor  is  financially  and 
technically  prepared  to  do  his  part  of  the  job.  A major  pitfall  is  to 
skimp  on  reviews  in  the  mistaken  belief  that  this  will  save  the  govern- 
ment money  and  schedule.  Experience  has  shown  otherwise. 


In  planning  the  review  or  audit  with  your  contractor,  request  that 
he  bring  people  who  can  define  the  situation  and  deal  with  problems, 
otherwise  much  time  (and  money)  will  be  wasted. 

3.2.2  Location  of  Reviews  and  Audits 

Hold  review  and  audit  meetings  where  the  needed  information  is 
located.  In  most  cases,  this  means  at  the  contractor's  facility,  and, 
unless  you  otherwise  specify  in  the  contract,  they  must  be  held  there. 

There  may  be  some  complications  if  the  software  is  being  developed 
by  a subcontractor  and  the  prime  contractor  wishes  to  hold  the  review 
at  his  own  facility.  If  the  software  is  very  complex  it  is  better  for  the 
government  if  at  least  the  PDR  and  CDR  are  held  at  the  software  sub- 
contractor's facility.  Try  to  make  the  decision  on  subcontractor  parti- 
cipation prior  to  finalization  of  pertinent  contracts  (i.e.  , SOW's  and 
CDRL's).  Get  the  contractor's  commitment. 

3.2.3  Conduct  of  Reviews  and  Audits 

As  previously  mentioned  and  as  shown  graphically  in  Figure  3-1, 
a review  does  not  consist  of  a review  meeting  only.  It  is  a mini-project 
and  should  be  managed  accordingly.  The  following  elements,  shown  in 
Figure  3-1,  are  essential  parts  of  the  review  process. 

• The  plan,  which  includes  the  schedule,  the  material  to  be 
reviewed,  the  objectives  of  the  review,  the  participants  and 
their  responsibilities,  and  the  means  of  following  up  on 
action  items. 

• The  schedule  and  agenda,  published  and  distributed  to  all 
participants  at  least  three  times: 

1.  as  early  as  possible  to  let  them  know  what  is  expected; 

2.  just  before  their  action  is  needed,  as  a reminder;  and 

3.  with  the  material  to  be  reviewed. 

• The  pre-review/audit  planning  meetings  with  your  contractor 
(to  work  out  details)  and  with  your  own  review  team  (to  ensure 
that  all  bases  are  covered  and  all  participants  know  their  jobs). 
These  meetings  should  cover  the  topics  to  be  discussed  at  the 
review  and  administrative  details,  and  the  follow-up  plan. 


• Distribution  of  review/avdit  material  supplied  by  the  con- 
tractor. You  must  decide  if  the  contractor  should  handle 
the  distribution  or  should  send  copies  to  you  for  distribution. 
The  former  is  faster,  but  you  should  follow  up.  Action 
requests  should  be  sent  to  the  reviewers  identifying  the 
material  they  are  responsible  for  reviewing  and  should  include 
a form  for  their  response. 

• Collection  of  comments  from  your  reviewers.  These  should  be 
consolidated  before  the  review/audit  meeting  (or  meetings). 

Table  3-2  is  a summary  of  the  personnel  anc.  actions  associated 
with  pre -meeting  activities. 


If  the  preparations  have  been  adequate,  the  review  meeting  will  be 
productive  and  will  achieve  its  goals.  The  meeting  normally  consists  of 
presentations  by  the  contractor  personnel,  followed  by  questions  and 
answers.  Side  meetings  may  be  held  work  any  special  technical  issues 
and,  perhaps,  detailed  interchange  on  necessary  changes  to  draft  docu- 
mentation. Agreements  and  action  items  close  the  meeting. 

NOTE:  It  is  important  to  obtain  co- signed  minutes  containing 
written  agreements,  action  items,  etc.  before  the  review/audit  ends. 

If  it  runs  for  several  days,  they  should  be  obtained  on  a daily  basis; 
otherwise,  there  will  be  confusion  on  the  last  day  about  agreement  on 
specific  points. 

Some  general  points  to  bear  in  mind  for  review  meetings  are 
the  following: 

s Keep  proper  records  of  the  meeting  (see  Section  4. 1. 3.4 
of  MIL-STD-1521A);  all  action  items  should  be  in  writing, 
clearly  assigned  to  an  organization,  and  have  due  dates. 

s Avoid  verbal  agreements  without  formal  documentation; 
contractual  implications  must  be  clearly  understood  in 
each  case;  otherwise,  the  agreement  may  be  forgotten 
or  misconstrued,  the  contractor  may  believe  that  he  has 
received  technical  direction  when  you  had  no  such  intent, 
and,  your  technical  support  may  be  pre-empting  your 
responsibility  (review  all  data  communicated  to  the  con- 
tractor). Obtain  assistance  (e.  g. , PCO,  ACO)  for  potential 
contractual  implications. 
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Figure  3-1.  Review/Audit  Process 
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Table  3-2.  Content  of  Pre -Review /Audit  Milestones 


Milestone 

Personnel  Involved 

Agenda  /Actions 

Preliminary  Planning  Meeting 
with  contractor 

USAF  Protect  Engineer 

USAF  Review  Co-Chairman 

Contractor  Project  Manager 

Contractor  Review  Chairman 

1 

1 

i 

i 

1 

1.  Review  objective* 

2.  Review  schedule 

3.  Review  location 

4.  Documents  to  be  distributed 
prior  to  review 

5a  Agenda  for  review  meeting 

6.  Handling  of  review  comments 

7.  Action  item  system 

8.  Approvals  and  authorizatio.  * 
expected  from  government 

Negotiate  with  agencies 
for  people 

USAF  Project  Engineer 

Agency  Management 

Agency  technical  groups 
whose  support  you  seek 

1.  Background  of  project 

2.  Relevance  to  agency 

3.  Project  review  needs 

4.  Specific  skills  or  names 
you  want  and  why 

5.  Commitment  of  names  to  a 
schedule 

Prepare  plan  for  review  /audit 

USAF  and  Contractor  Co- 
Chairman 

Issue  plan  with  objectives* 
schedules*  document 
descriptions 

Designate  assignments 
to  members  of  review/ 
audit  team 

USAF  Review  Co-Chairman 

USAF  Project  Manager 

Selected  reviewers  if  needed 

1.  Define  government  objectives 

2.  Define  how  review  will  meet 
objectives 

3a  Assign  tasks  to  reviewers 
(what  to  read*  what  to 
comment  on). 

Final  Planning  Meet 
with  contractor 

USAF  and  Contractor  Co- 
Chairman 

1.  Final  review  schedule 

2.  Final  review  document  list 

3a  Final  meeting  agenda 

4.  Documents  for  government 
signature  (actual  documents, 
or  specimens). 

Issue  Review  Notice  (con- 
tractor prepared  Agenda 
is  an  attachmcntl 

USAF  Co-Chairman 

Issue  notice  of  review  from 
government  program  office. 

Meet  with  your  review/audit 
team,  pl.in,  assign 

USAF  Co-Chairman 

All  government  reviewers 

1.  Review  objectives 

2.  Material  to  review  and  schedule 

3.  How  comments  are  to  be  written; 
when  due 

4.  List  of  review  tasks  and  assign- 
ments 

5.  Ground  rules  and  forbidden 
topics 

Comments  in  from 
reviewers 

USAF  Project  Engineer 

USAF  Co-Chairman 

1.  Tabulate  all  comments  into 
one  document 

2.  Prioritise 

)•  Send  to  contractor  prior  to 
review  meeting 
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• Do  not  let  the  review  meeting  get  out  of  control;  brief  your 
people  on  the  objectives  of  the  project.  Define  "forbidden" 
areas  (such  as  discussing  design  before  the  requirements 
are  defined);  ask  the  contractor  to  do  the  same. 

• Do  not  let  the  contractor  keep  you  from  finding  out  what  is 
going  on  with  subcontractors,  suppliers,  and  vendors.  If 
he  takes  the  position  that  it  is  his  responsibility,  gently 
remind  him  that  it  is  your  responsibility  to  see  that  the 
government's  interests  are  being  served. 

• Ask  each  reviewer  to  summarize  for  you  in  advance  how 
he  will  know  whether  or  not  all  applicable  specifications 
and  regulations  have  been  met.  If  your  project  is  too 
small  to  have  individual  reviewers  do  thi3,  do  it  yourself. 

3.2.4  Closing  Out  Reviews  and  Audits 

A review  does  not  end  when  the  flip  charts  are  folded  and  the  crowd 
leaves.  Most  reviews  uncover  discrepancies  and  problems,  and  some 
action  is  required  for  each  one.  Only  when  the  action  is  satisfactorily 
assigned  and  the  response  evaluated  as  satisfactory  can  the  review  be 
considered  closed  out.  In  general,  the  government  is  expected  to  evaluate 
work  done  and  authorize  future  work  and  provide  technical  direction  as 
required.  The  outcome  of  a review  of  particular  item  can  be 

1.  review  fully  acceptable; 

2.  review  acceptable  with  discrepancies  noted  and  action  items 
covering  discrepancies  accepted  by  the  contractor;  and 

3.  review  unacceptable,  the  contractor  told  to  try  again  (in  this 
case,  the  schedule  must  be  revised  and  a date  for  a new 
review  determined).  Rescheduling  should  be  documented 

in  the  minutes. 

For  each  discrepancy  noted  in  case  2,  one  of  the  following  actions 
is  required. 

1.  Rejected  (you  and  the  contractor  must  agree). 

2.  Deferred  to  a future  review  (such  action  is  often  taken 
when  a comment  is  valid  but  premature,  such  as  a 
comment  on  a design  concept  made  at  a review  of 
requirements). 

3.  A solution  is  generated;  it  is  then  reviewed  and  accepted 
(or  iterated;  if  necessary)  by  the  contractor  and  the  govern- 
ment program  office  and  suitably  documented. 
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A particular  type  of  discrepancy  to  be  wary  of  is  the  "TBD"  (to 
be  determined)  i.e. , some  value  to  be  determined  later.  Keep  track  of 
these;  assign  action  items  with  dates  for  the  TBD's  to  be  converted  into 
real  data. 

Be  sure  your  contractor  has  an  action  items  system  to  record 
agreements  and  verify  follow-up.  ^ Assign  someone  to  follow  uRon  action 
items  status;  the  review  is  not  closed  out  until  all  action  item  responses 
have  been  evaluated  to  the  government's  satisfaction  and  items  formally 
closed  out. 

'Review  both  the  contract  and  the  project  plan  to  see  if  any 
agreements  reached  at  the  review  require  modification  to  either  docu- 
ment. Even  when  changes  are  financially  in  scope,  they  may  require 
a contract  change.  (For  example,  the  contract  calls  for  use  of  SIN/COS 
routine  X;  agreement  made  to  use  routine  Y at  no  cost  to  the  govern- 
ment, but  the  provisions  of  the  contract  have  changed  and  need  updating, 
even  though  it  has  no  cost  impact). 

The  final  step  in  any  review  or  audit  is  to  evaluate  the  work 
completed  and  to  authorize  work  on  the  next  phase  of  the  project.  If 
the  review/audit  is  judged  unsatisfactory  it  may  be  necessary  to  resched- 
ule the  review  or  audit  for  a later  date  when  the  contractor  will  be  ready. 

If  an  additional  formal  review  is  not  required,  it  may  be 
advisable  to  hold  a meeting  to  close  out  action  items.  This  procedure 
is  especially  important  where  work  has  been  accepted  contingent  upon 
the  completion  of  a number  of  major  action  items. 

The  responsibilities  of  the  government  in  reviews  and  audits  are: 

• Evaluate  work  performed  (documents  and  informal  data), 

• Provide  technical  direction  as  necessary. 


*If  you  put  MIL-S-52779  (AD)  on  the  contract,  you  will  require  this 
as  part  of  the  contractor's  QA  activity. 
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• Approve  review/audit  official  minutes  (after  iteration 
as  required) 

• Written  acknowledgement  that  review /audit  was 
accomplished,  and 

e Followup  to  closeout  of  action  items. 

3.3  OTHER  GENERAL  GUIDELINES 
3.3.1  Some  Effective  Reviewing  Techniques 

There  are  several  techniques  that  can  increase  the  effectiveness 
of  reviews;  in  most  cases,  all  these  techniques  can  be  applied.  It  is  a 
good  idea  to  go  over  the  techniques  with  your  reviewers  as  a means  of 
helping  them  give  you  their  best.  The  techniques  to  be  discussed  are: 

e assigning  reviewers'  points  of  view, 

• outlining  before  opening, 

• tracing  a stimulus -response  chain  through  the  software, 

• decision-oriented  reviewing,  and 

• commenting  on  reviews  in  writing  and  in  terms  of 
specifics  rather  than  general  satisfaction  or 
dissatisfaction. 

Assigning  Reviewers'  Points  of  View.  An  easy  way  to  manage  a review 
is  to  give  all  of  the  review  material  to  all  reviewers  and  let  them  read  as 
they  choose.  It  is  a bad  way  to  manage  a review.  Some  topics  will  be 
covered  too  much;  some,  not  enough,  or  not  at  all.  The  sheer  volume  of 
the  material  may  cause  reviewers  to  browse  instead  of  reading  carefully. 

The  solution  is  to  assign  different  "points -of- view"  to  different 
reviewers.  Each  reviewer  can  then  carefully  select  the  portions  he 
needs  to  review.  Typical  points  of  view  that  can  be  assigned  to 
reviewers  are: 

e feasibility  (Can  it  be  built  within  budget,  on  schedule, 
and  run  on  the  selected  machine?  ), 

' - .A  f ; , . ..  . .v  / 


/ 


• interfaces  (Do  we  understand  correctly  just  how  the 
software  connects  to  the  rest  of  the  system?). 


e operability  (Will  it  really  do  the  job  the  Air  Force 
needs  done?), 

e supportability  (Can  it  be  operated  and  maintained 
by  the  kinds  of  personnel  proposed?),  and 

e adversary  (Are  there  deficiencies  or  weak  spots  that 
can  be  exploited  by  an  adversary?). 

Outlining  Before  Opening.  The  general  idea  of  this  technique  is  to  check 
for  completeness  and  to  keep  you  in  command  of  the  review  material. 

The  process  begins  when  you  get  your  reviewing  assignment  (or  lay 
out  your  review  plan).  At  that  point  you  decide,  a priori,  what  you 
expect  to  see  in  terms  of: 

e topics  covered  and  topics  omitted, 

• depth  of  treatment  (e.  g. , flowcharts,  equations,  final 
code), 

• pages  per  topic, 

• problems  you  expect  to  see  defined  and  dealt  with. 
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• topics  on  which  you  expect  no  problems, 
a type  of  document  (first-draft,  engineering,  final). 


• coherency  of  document  (e.g. , all  software  packages 
consistent,  or  is  some  inconsistency  to  be  expected  at 
this  point),  and 

• traceability  of  requirements  (should  they  be  traceable  at 
this  point). 

Now  when  you  begin  to  read  the  material,  you  will  be  checking 
against  a set  of  expectation?.  These  expectations  keep  you  alert  and 
give  you  a basis  for  judgment. 

Tracing  Stimulus-Response.  A common  method  of  analyzing  avionics 
software  is  by  function:  sensor  input,  alignment  and  calibration. 


instrument  compensation,  navigation,  and  fire  control.  An  alternate 
technique  is  to  examine  the  software  in  terms  of  stimuli  input  to  the 
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my  stem  and  responses  generated  by  the  system.  For  example,  you  might 
take  a radar  tracking  return  and  follow  it  through  the  software  until  it 
results  in  some  externally  visible  action  such  as  a weapons  release 
command.  This  technique  is  effective  for  analysing  avionics  software 
because  the  major  function  of  such  software  is  to  close  control  loops 
either  directly  or  through  the  pilot. 


A complete  check  (desireable  for  critical  software)  would  mean 
examining  all  input  stimuli  and  tracing  them  through  to  outputs,  but  this 
completeness  may  be  too  time-consuming  and  expensive  tor  a review. 
An  alternative  approach  is  to  select  four  to  six  inputs  and  trace  them; 
half  should  be  from  the  most  critical  loops,  with  the  other  half  chosen  at 
random.  If  you  find  problems  with  this  sample,  more  analysis  is 
indicated.  The  types  of  problems  likely  to  be  uncovered  with  this 
technique  are: 


e undefined  actions  for  special  cases, 

• incorrect  responses  under  some  (or  maybe  all)  conditions, 

• inputs  never  used,  and 

• outputs  never  generated. 

Decision-Oriented  Reviewing.  An  excellent  way  to  ensure  the  value  of 
a review  is  to  orient  it  toward  the  decisions  that  must  result.  First, 
list  the  decisions  that  you  will  have  to  make  after  the  review  (e.g., 
authorization  to  code).  For  each  decision,  define  what  you  need  to  know 
to  reach  that  decision  (e.g.,  are  there  plans  and  standards  for  coding.  . . 
is  the  design  complete.  . .).  Then  review  the  material  (or  structure  the 
review)  to  produce  the  information  needed. 

One  advantage  of  this  approach  is  that  it  avoids  the  embarrassment 
of  finishing  a review  and  finding  that  you  did  not  learn  what  you  needed  to 
support  a logical  decision.  Another  is  that  it  is  a test  of  relevance, 
allowing  you  to  avoid  topics  that  are  useless  to  the  purpose  of  the  review. 
Table  1-2  (Section  1)  provides  capsule  summaries  of  the  purposes,  and 
hence  the  decisions  required,  for  each  type  of  review. 
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Commenting  in  Writing.  A good  set  of  written  comments  can  be  one  of 
the  most  valuable  products  of  a review;  conversely,  one  of  the  least 
useful  products  of  a review  is  unrecorded  comments  or  worthless 
written  comments.  Your  reviewers  should  have  some  guidelines  on 
writing  comments. 

• All  comments  must  be  written,  and  the  person  making  the 
comment  is  responsible  for  putting  it  in  writing  (unless 
that  action  is  specifically  assigned  to  someone  else). 

• A comment  must  clearly  identify  the  problem,  as  opposed 
to  stating  only  a proposed  solution;  any  solutions  proposed 
must  be  clearly  separated  from  the  problem. 

• A problem  must  relate  to  some  defined  system  issue. 

For  example,  the  comment  "has  inadequate  comments 
in  the  code"  is  not  acceptable;  it  should  read  "violates 
the  requirements  for  code  commenting  standards  in 
Section  3.6.1  of  the  project  plan"  or  "violates  the 
requirements  for  maintainability  in  Section  4.  7 of  the 
statement  of  work.  " 

• Comments  should  be  ranked  in  some  order  of  importance 
either  by  ranking  directly  or  by  some  scoring  scheme 
such  as:  Level  1,  violates  contract  or  will  cause  system 
failure;  Level  2,  results  in  marginal  system  performance; 
Level  3,  results  in  suboptimal  system  performance; 

Level  4,  other. 

Unless  you,  the  project  engineer /manager  in  charge  of  the  review, 
create  such  guidance  for  your  reviewers  you  may  be  inundated  by  com- 
ments that  you  cannot  use  or  that  are  irrelevant. 

3.3.2  Selecting  Reviewers 

Many  reviews  fail  before  they  start  simply  because  the  project 
engineer  did  not  devote  enough  effort  to  lining  up  the  right  reviewers. 
Poorly  chosen  reviewers  can  actually  decrease  the  effectiveness  of 
review  team.  Thus,  a few  extra  hours  s^ent  at  the  beginning  getting 
the  right  people  has  great  leverage  for  iownstream  performance. 
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The  three  basic  considerations  in  selecting  reviewers  are 

• determining  the  technical  skills  needed  to  conduct 
the  revi°w. 


• finding  people  who  know  how  to  review  and  will  spend 
the  time,  and 

• identifying  people  who  must  be  invited  for  information 
or  concurrence. 


Technical  Skills.  Study  the  objectives  of  the  review/audit  and  then 
decide  what  skills  are  needed.  You  have  seven  main  sources  of  skills: 


1.  your  own  program  office, 

2.  supporting /using  agencies  (e.  g.  , TAC,  SAC,  AFLC), 

3.  your  internal  engineering  organization  (e.g.  , ASD/EN 
SAMSO/AW,  and  SAMSO/YC), 


4.  Air  Force  laboratories 


5.  SETA  agency  (Aerospace,  Mitre,  etc.) 

6.  SETA  contractor  (if  there  is  one),  and 


Review  Capabilities.  You  cannot  always  control  reviewer  selection, 
and  ideal  reviewers  are  not  always  available.  You  should  try  to  get  at 
least  a few  key  people  who  are  good  reviewers.  A good  reviewer  will 

• seo  how  well  it  will  work  as  designed,  not  how  he 
would  design  it; 

• take  clear  positions  and  stand  by  them; 

• conduct  himself  professionally  and  inspire  trust  (other- 
wise, contractor  personnel  may  withhold  information 
for  fear  of  personal  consequencies); 

e focus  on  identifying  problems,  not  on  solving  them  in  real 
time  during  the  re\iew  meeting; 

e be  capable  of  identifying  errors  of  omission  as  well  as  of 
commission;  and 


e write  pertinent  comments 


4.  GUIDANCE  FOR  INDIVIDUAL  REVIEWS  AND  AUDITS 
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A series  of  technical  reviews  and  configuration  management  audits 
should  be  scheduled  and  conducted  at  meaningful  points  (corresponding 
with  forma)  "baselines"  and  intermediate  milestones,  see  Table  1-1)  in 
the  CPC  I acquisition  process  to  permit  assessment  of  progress  and  pre- 
pare for  the  next  development  step  and  to  establish  new  baseline  config- 
uration identifications  for  the  product.  This  section  discusses  the  seven 
reviews  and  audits  specified  by  MIL-5TD-1521A  (USAF)  and  an  additional 
informal  (contractor  internal)  Test  Readiness  Review  (TRR),  see  Table  1-1 
and  subsection  4.5,  which  corresponds  to  a meaningful  intermediate  mile- 
stone. For  each  airborne  systems  acquisition,  the  specific  number,  con- 
tent and  scope,  and  conduct  of  the  reviews  and  audits  should  be  included  in 
the  governing  contractual  documentation  (SOW  and  CDRL)  to  assure  con- 
tractor committment  of  adequate  technical  and  financial  resources  to 
support  meaningful  reviews  and  audits. 

The  technical  reviews  (SSR,  SDR,  PDR,  CDR,  TRR)  are  primarily 
systems  engineering  or  design  oriented  and  focus  on  CPC  I requirements 
definition/allocation,  design  and  test  preparation.^  The  audits  (FCA,  PCA 
and  FQR)  are  primarily  configuration  management  oriented  and  focus  on 
CPCI  performance  qualification  and  configuration  identification  verification. 
Time  phasing  of  the  reviews  and  audits  versus  system  and  CPCI  life  cycle 
activities  is  shown  in  Figures  1-1  and  1-2. 

General  guidance  in  planning,  preparing,  and  conducting  reviews  and 
audits  was  provided  in  Section  3 (e.g. , responsibilities  of  the  participating 
organizations,  specific  techniques  for  effective  reviews  and  audits,  etc.) 
This  section  provides  detailed  guidance  for  each  review  and  audit  to  the 
Air  Force  Project  Office  and  engineering  personnel  responsible  for  the 
acquisition  of  CPCI's  for  airborne  systems.  The  approach  in  subsections 
4.1  through  4.8  is  to  provide  a summary  "checklist"  table  accompanied 
by  a brief  narrative  highlighting  leverage  issues,  unique  preparations,  etc. 
for  each  individual  review  and  audit. 


^Throughout  this  guidebook,  reference  to  a CPCI  Development  Specification 
includes  the  interface  and  data  requirements  specification  if  these  volumes 
are  produced  separately. 


4.1  SYSTEM  REQUIREMENTS  REVIEW  (SRR) 

System  Requirements  Reviews  (SRR's)  are  also  called  system/ 
system  segment  requirements  reviews.  SRR's  are  in-process  reviews 
which  may  be  conducted  any  time  consistent  with  contract  provisions; 
however,  they  are  usually  conducted  on  concept  definition  contracts  or 
early  in  concept  validation  contracts  for  a new  large-scale  system. 

Table  4-1  provides  a summary  of  SRR. 

4.1.1  Leverage  Issues 

The  primary  objective  of  this  review  is  to  evaluate  and  approve  the 
allocation  of  system/ system  segment  requirements  against  validated 
mission  requirements.  Another  major  objective  is  to  evaluate  the  pro- 
gress of  the  systems  engineering  analyses/studies  and  design  synthesis 
efforts  toward  convergence  to  an  optimum  system/subsystem  configuration. 

r 

From  the  viewpoint  of  eventual  CPCI  acquisition,  the  major  objective 
is  to  evaluate  the  preliminary  approach  to  allocating  system/subsystem 
requirements  to  embedded  computer  resources  and  validating  the  appli- 
cation; e.  g. , 

9 Requirements  definitive  and  unambiguous  ? How  many 
"TBD's ''  ? 

• Technically  feasible?  For  example,  interface  rates 
versus  "real  time"  constraints.  High  risk  elements? 

• Number  and  complexity  of  interfaces? 

Other  considerations  include;  responsiveness  to  SOW,  traceability 
to  requirements  (possible  contractor  gold  plating),  and  testability  of 
requirements. 

Satisfactory  completion  of  SRR  and  formal  customer  approval  of  the 
System/System  Segment  Specification  establishes  the  system/system 
segment  Functional  Baseline. 


4.1.2  SRR  Post -Review  Action 


After  completion  of  the  SRR  the  contractor  is  responsible  for 
publishing  and  distributing  the  official  (co-signed  by  customer  and 
contractor)  SRR  minutes.  The  minutes  should  clearly  record  all  agree- 
ments and  all  action  items,  including  suspense  dates,  and  assign  specific 
responsibility  to  the  Procuring  Activity  and/or  the  contractor.  The 
Procuring  Activity  provides  formal  acknowledgement  to  the  contractor 
of  the  accomplishment  of  the  SRR  after  receipt  of  SRR  minutes. 

4.2  SYSTEM  DESIGN  REVIEW  (SDR) 

The  System  Design  Review  (SDR)  evaluates  the  system/system 
segment  synthesis,  including  the  supporting  systems  engineering  analyses 
and  studies,  resulting  in  the  allocation  of  system/system  segment 
requirements  (System/System  Segment  Specification)  to  individual  equip- 
ment configuration  items  (Cl's)  and  computer  program  configuration  items 
(CPCI's),  to  establish  a requirements  baseline  as  reflected  in  a Part  I 
Development  Specification  for  each  Cl  and  CPCI.  SDR  is  usually  the  final 
review  prior  to  submittal  of  the  Validation  Phase  products  or  as  the 
initial  review  in  the  Full-Scale  Development  Phase  for  systems /subsystems 
not  requiring  a formal  Validation  Phase.  Table  4-2  provides  a summary 
of  SDR. 

4.2.1  Leverage  Issues 

The  Procuring  Activity  should  evaluate  any  changes  to  the  System/ 
System  Segment  Specification  (Functional  Baseline)  for  consistency  with 
validated  mission  requirements  and  ensure  that  all  System/System  Segment 
Specification  requirements,  including  performance  and  test  requirements, 
are  optimally  assigned,  and  traceable  to.  Cl's  and  CPCI's. 

t 
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The  developxnent/8upport  concept  for  each  CPCI,  as  specified  in 
the  Computer  Program  Development  Plan,  should  be  carefully  evaluated 
to  ensure: 

• completeness  (nothing  has  been  overlooked) 

• technical /management  adequacy,  and 

• schedules  and  projected  costs  are  realistic. 

The  key  decision  at  SDR,  associated  with  CPCI  acquisition,  is  approval * 
of  the  CPCI  Part  I Development  Specification.  Requirements  problems 
(inconsistent  requirements,  incomplete  requirements,  missing  require- 
ments, over-constraining  requirements,  incorrect  requirements,  etc.) 
not  detected  in  the  approved  Part  I Development  Specification  may  not 
surface  until  very  late  in  the  acquisition  cycle  (e.g. , integrated  systems 
testing  or  operational  use)  with  resulting  major  re-werk  required  and 
corresponding  cost  and  schedule  impacts.  The  Procuring  Activity  should 

• ensure  that  Part  I Development  reflects  an  understanding 
of  the  operational  mission; 

• require  that  the  contractor  produce  analyses  demonstrating 
the  completeness,  feasibility  and  testability  of  the  require- 
ments set  and  consistency  with  system/subsystem  require- 
ments and  external  interfaces; 

• analyze  each  individual  requirement  to  verify 

• proper  and  clear  statement  which  distinguishes  between 
mandatory  requirements  and  design  goals  or  options, 

a compatibility  with  system  level  objectives,  where 
appropriate. 


• technical  feasibility  and  risk 

pjg 

Satisfactory  completion  of  the  SDR  plus  formal  customer  approval 
(authentication)  of  a CPCI  Part  1 Development  Specification  establishes 
an  Allocated  Baseline  for  the  CPCI  as  the  basis  for  the  ensuing  prelim- 
inary design  effort.  The  objective  is  to  review  a complete  draft  of  the 
Part  I Development  Specification  at  SDR  and  make  any  required  changes 
as  part  of  the  SDR  action  item  close-out  process  permitting  an  approved 
(authenticated)  Development  Specification  as  soon  as  possible  after  SDR. 
The  risk  associated  with  delaying  the  authentication  of  the  Development 
Specification  until  PDR,  as  has  been  done  on  some  projects,  is  the 
potential  compromise  of  some  or  all  of  the  CPCI  preliminary  design 
effort. 
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• testability. 

• completeness  (i.  e. , identify  TBD's,  explicit  or  implicit),  and 
e identification  of  any  necessary  design  constraints. 

e require  that  the  contractor  justify  all  "not  applicable" 
and  explain  all  "to  be  determined"  entries  including  an 
approach  to  resolve  them; 

• ensure  that  the  allocated  irterfaces  are  explicitly  identified 
and  detailed  in  terms  of  message  formats,  update  rates,  etc.; 

• determine  the  compatibility  of  the  requirements  set  with 
contract  schedule  and  funding,  and  other  project  resources 
(personnel,  facilities,  etc. )}' 

• ensure  every  requirement  will  be  tested  (i.e.,  Development 
Specification,  Section  4 accounts  for  every  requirement  in 
Section  3);  and 

• ensure  that  file  total  requirement  set  provides  an  adequate 
basis  to  begin  preliminary  design. 

4.2.2  SDR  Post -Review  Action 

After  completion  of  the  SDR,  the  contractor  is  responsible  for 
publishing  and  distributing  the  official  (co-signed  by  customer  and  contractor) 
SDR  minutes.  The  minutes  should  record  all  agreements  and  all  action 
items,  including  suspense  dates,  and  assign  specific  responsibility  to  the 
Procuring  Activity  and/or  the  contractor.  The  Procuring  Activity  provides 
formal  acknowledgement  to  the  contractor  of  the  accomplishment  of  the 
SDR  after  receipt  of  the  SDR  minutes. 

4.3  PRELIMINARY  DESIGN  REVIEW  (PDR) 

The  PDR  is  a formal  review  of  the  basic  design  concept  for  a CPCI, 
which  establishes  a preliminary  design  approach  and  the  implementation 
and  test  plans  necessary  to  proceed  into  detailed  design  and  development. 
Table  4-3  provides  a summary  of  PDR. 

4.3.1  Leverage  Issues 

The  key  decision  at  the  PDR  for  a CPCI  is  to  determine  if  the  con- 
tractor's basic  design  approach  and  associated  implementation  and  test 
planning  provide  an  adequate  basis  for  proceeding  to  detailed  design. 
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The  following  actions  should  be  considered  by  the  Procuring  Activity. 

a.  Require  the  contractor  to  demonstrate  that  every  requirement 
in  the  Part  1 Development  Specification  (including  approved 
ECP's)  has  been  properly  accounted  for  (and  is  traceable) 

in  the  dessgn. 

b.  Ensure  that  the  design  is  valid  (complete,  consistent, 
feasible,  maintainable,  and  testable). 

c.  Require  the  contractor  to  demonstrate  that  the  aggregate 
design  budgets  (e.g.,  storage,  timing,  accuracy)  satisfy 
the  Part  I Development  Specification,  and  additionally, 
do  not  exceed  the  limitations  of  the  CPCl's  physical  and 
functional  environments. 

d.  Evaluate  the  adequacy  of  design  tradeoff  studies  and  pre- 
liminary performance  estimates  substantiating  baric 
design  approach  and  algorithm  selection.  Identify  high 
risk  areas,  if  any,  and  approach  to  risk  reduction. 

e.  Ensure  adequate  interface  definition  with  equipment  Cl's  and 
other  CPCl's,  including  corresponding  test  requirements. 

f.  Ensure  Part  I Development  Specification  is  adequate  and 
complete.  Evaluate  any  proposed /approved  changes  to 
previously  authenticated  version. 

g.  Review  implementation  planning  (e.  g. , required  tools  and 
facilities)  and  test  planning  for  adequacy  and  completeness. 

h.  Identify  any  open  issues  of  a technical  or  contractual  nature, 

(e.g.,  any  requirements  not  satisfied)  including  disposition  and/ 
or  approach  to  resolve  the  issues  documented  in  the  PDR  minutes). 

4.3.2  PDR  Post-Review  Action 

^ — — — — ^ 

After  completion  of  the  PDR,  the  contractor  is  responsible  for 
publishing  the  official  (co-signed  by  customer  and  contractor)  PDR  minutes. 
The  minutes  should  clearly  record  all  agreements  and  all  action  items, 
including  suspense  dates,  and  assign  specific  responsibility  to  the  Procuring 
Activity  and/or  the  contractor.  The  Procuring  Activity  provides  formal 
acknowledgement  to  the  contractor  of  the  accomplishment  of  the  PDR  after 
receipt  of  the  PDR  minutes,  but  should  not  formally  "approve"  the 
preliminary  design. 
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Table  4-3.  Summary:  Preliminary  Deaig 
Review  (PDR):  Acquisition  of 
CPCPs  for  Airborne  Systems 

PDR 

Purpose 

Items  to  be  Reviewed 

Formal  Documentation 

Other  Technical /Management  Data 
(Contractor  Presentations, 

Internal  Reports,  etc.) 

• The  PDR  for  sack  CPCI 
(may  be  grouped)  ie 
held  early  in  the  Full 
Scale  Development 

Phaee  (between  SDR  and 
CDR)  when  sufficient 
design  analysis  has 
been  accomplished  to 
arrive  at  a computer 
program  architecture 
and  overall  modular 
structure  which  will 
provide  the  basis  for 
detailed  design. 

• Availability  of  an  auth- 
enticated Part  1 Devel- 
opment Specification  is 
a prerequisite  for  any 
CPCI  to  be  PDR'd. 

• Evaluate  the  basic  design  approach  for  Completeness, 
adequacy  and  compatilility  with  allocated  require- 
ments (Part  I Development  Specification),  e.g.. 

• Ensure  compatibility  of  the  design  approach  with 
the  Part  I Development  Specification. 

e Evaluate  the  progress,  technical  adequacy  and 
risk  resolution  (on  a technical,  cost,  and 
schedule  basis)  of  the  selected  design  approach. 

• For  each  CPCI.  establish  the  existence  and  com- 
patibility of  the  physical  and  functional  interfaces 
between  the  CPCI,  other  CPCI's,  hardware  Cl's, 
and  facilities. 

• Review  all  changes  to  the  System/System  Segment  spe- 
cification and  Part  I Development  Specification  to  ensure 
that  they  are  properly  incorpo rated >in  the  basic  design 
approach,  the  draft  Part  II  Product  Specification,  and 
test  planning. 

• Review  status  of  all  negative  and  provisional  entries 
such  as  "not  applicable " (N/A)  or  "to  be  determined" 
(TBD)  in  Section  4 of  the  System/System  Segment 
Specification  and  Part  I Development  Specification. 
Review  all  positive  entries  'or  technical  adequacy. 

e Review  all  detailed  functional  interfaces,  and  corres- 
ponding test  requirements,  with  hardware  Cl's  and  with 
other  CPCI's.  Review  word  lengths,  meesage  formats, 
transfer  rates,  timing,  storage  implications,  etc.  At 
this  time,  applicable  interfaces  between  a CPCI  and 
system  hardware  Cl's  should  be  sufficiently  defined  to 
permit  CPCI  design  to  proceed  independently. 

• Review  the  CPCI  interactions  with  Human  Factor 
requirements.  Review  all  man-machine  interfaces 
for  feasibility,  adequacy  and  completeness. 

e Review /evaluate  the  overall  structure  of  the  CPCI  for 
completeness  and  adequacy,  with  emphasis  on  the 
following: 

e Allocation  of  Computer  Program  Components 
(CPC's)  to  the  functions  (requirements)  deline- 
ated in  the  Part  I Development  Specification  and 
functional  flows. 

e Storage  requirements  and  allocation. 

• Computer  program  operating  sequences. 

s Design  of  the  data  base. 

e Analyse  critical  timing  requirements  of  the  system  as 
they  apply  to  the  CPCI  to  ensure  that  the  proposed  CPCI 
design  approach  satisfies  the  timing  requirements. 
Review  execution  time  estimates  for  reasonableness 
and  compatibility  with  timing  requirements. 

• Review  interface  test  requirements  specified  in  Sec- 
tion 4 of  the  development  specification  for  compatibil- 
ity, currency,  technical  adequacy,  elimination  of  redun- 
dant test.  Ensure  that  all  associated  test  documents 
reflect  these  interface  requirements. 

• Review  test  planning  documentation  to  ensure  that  the 
test  program  satisfies  the  test  requirements  specified 
in  Section  4 of  the  System/System  Segment  Specifica- 
tion and  Section  4 of  the  Part  I Development  Specifica- 
tion. 

e Ensure  that  all  test  planning  documentation  has  been 
updated  to  include  any  new  test  support  requirements. 

e.  Updated  System /System 
Segment  Specification 
(if  necessary). 

• Final  Part  I Develop- 
ment Specification  for 
each  CPCI. 

s Partial  Part  II  Product 
Specification  for  each 

CPCI  missing  only  the 
detailed  component  level 
flow  charts;  "build  to" 
flows  will  be  in  CDR 
drsft,  "as  built"  flows 
will  be  in  final  version 
prior  to  PC  A. 

e Preliminary  Qualification 
(Acceptance)  Test  Plan 
for  each  CPCI. 

e Preliminary  Data  Base 
Document  for  each  CPCI. 

e CPCI  functional  flows  to  the  level  of  flow 
charting  that  identifies  allocation  of  Part  I 
Development  Specification  requirements  to 
individual  Computer  Program  Components 
(CPC's)  and  depicts  the  sequence  of  opera* 
tions  within  the  CPCI  and  within  a compute] 
program  component  at  least  to  a Part  I pro 
cessing  requirement  level. 

e Storage  allocation  charts  detailed  for  each 
CPCI  as  a whole,  describing  the  allocation 
of  available  storage  to  individual  Computer 
Program  Components  (CPC's).  Identifies* 
tion  of  timing  requirements,  sequencing 
requirements,  and  relevant  equipment  con* 
straints  used  in  determining  the  allocation 
should  be  provided. 

e Sising  and  timing  estimates  and  budgets. 

• Description  of  CPCI  control  functions 
including  the  executive  control  and  start/ 
recovery  features  for  the  computer  pro- 
gram system,  method  of  initiating  system 
operation,  and  features  that  permit  recov- 
ery from  system  malfunction. 

• Description  of  the  overall  hierarchial 
structure  of  each  CPCI  and  the  rationale 
for  the  indicated  functional  decomposition 
into  components,  routines,  etc. 

• Description  of  the  data  base  structure/ 
organisation  to  a level  that  identifies  data 
types  and  characteristics,  structure  lay- 
out, and  allocation  of  data  storage 

e Design  trade-off  studies,  e.g.,  synchro- 
nous, asynchronous,  or  hybrid  executive; 
algorithm  alternatives;  etc. 

• Preliminary  performance  estimates. 

• Interface  definition  between  the  CPCI, 
hardware  Cl's  and  other  CPCI's. 

• Identification  of  unique  security  require- 
ments, if  any,  and  a description  of  the 
techniques  to  be  used  for  satisfying  them. 

• Identification  of  any  reentrancy  require- 

ments and  a description  of  the  technique 
for  implementing  and  testing  reentrant  j 

routines. 

i 

e Description  of  the  availability,  adequacy, 
and  planned  utilisation  of  tools  and  facility 
for  CPCI  development,  including  System / 
CPCI  exercising.  ] 

e Description  of  any  special  simulation,  dab 
reduction,  or  utility  tools  that  are  not 
deliverable  under  terms  of  contract,  but 
which  are  planned  for  use  during  program 
development. 

• Status  of  all  ECP's  and  DPR'e  against 

each  CPCI.  ) 

*T1».  two  olo manta  that  dlatin^itah  "formal  baaellnaa'  from  "Intaimadiate  mUaatonoa"  aro  that  (1)  formal  baaalinaa  involve  formal  cuatomar  approval  of  conflgarat 
documentation  (epaclficationa),  and  (2)  approved  ■ pacification!  are  put  under  formal  configuration  control. 


Formal  Baeeline*Eatabliehed: 
N/A 


Intermediate  Mile  atone  t 
Preliminary  Doalgn  Approach 
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4.4  CRITICAL  DESIGN  REVIEW  (COR) 


The  Critical  Design  Review  (CDR)  is  a formal  review  conducted  on 
each  computer  program  component  (CPC)  of  the  CPCI  before  translating 
the  engineering  language,  logic,  and  algorithms  to  coded  instructions. 

The  CDR  ensures  that  the  detailed  design  solution  and  associated  imple- 
mentation plans  and  qualification/acceptance  test  planning  satisfies  the 
requirements  of  the  Part  I Development  Specification  and  establishes  the 
detailed  design  basis  for  the  CPCI. 

If  top-down  development  is  specified,  where  upper  levels  of  the  CPCI 
hierarchy  are  designed/developed  before  lower  levels,  a series  of  pro- 
gressive (incremental)  CDR's  is  required.  For  large,  complex  CPCl's, 
regardless  of  development  methodology,  incremental  CDR's  are  a common 
practice  to  review  logical  groupings  of  CPC'S.  Table  4-4  provides  a 
summary  of  CDR. 

4.4.1  Leverage  Issues 

The  key  decision  at  CDR  for  a CPCI  is  whether  or  not  the  contractor's 
detailed  design  baseline  and  associated  implementation  and  test  planning 
provide  an  adequate  basis  to  proceed  to  coding  and  testing  the  CPCI.  The 
Procuring  Activity  should: 

a.  Require  that  the  contractor  demonstrate  that  every  require- 
ment (Part  I Development  Specification)  has  been  properly 
accounted  for,  and  is  traceable  to,  the  detailed  design. 

b.  Ensure  that  the  detailed  design  is  valid  (complete,  con- 
sistent, feasible,  maintainable,  and  testable). 

c.  Ensure  that  the  detailed  design  and  critical  parameter 
budgets  (e.  g. , storage,  timing,  accuracy)  for  the  CPCl's 
do  not  collectively  exceed  the  limits  given  in  the  Part  I 
Development  Specification,  and  additionally,  do  not  exceed 
the  limitations  of  the  CPCI  physical  and  functional 
environments . 

d.  Evaluate  the  design  evaluation  and  tradeoff  studies  and 
performance  estimates  substantiating  the  detailed  design. 


e.  Review  current,  detailed  implementation  planning  and 
qualiiication/acceptance  teat  planning  for  adequacy  and 
completeness.  Approval  of  the  Qualification  (Acceptance) 

Test  Plan  provides  a controlled  definition  of  the  project's 
acceptance  test  program. 

f.  Identify  and  discuss  any  critical  issues,  e.g. , any 
requirements  not  satisfied,  and  provide  a resolution 
of  the  issues. 

g.  Identify  specific  computer  programming  ("build  to") 
documentation  which  will  be  released  for  coding  and  testing. 

4.4.2  CDR  Post-Review  Action 

After  completion  of  the  CDR,  the  contractor  is  responsible  for 
publishing  and  distributing  the  official  (co- signed  by  customer  and  con- 
tractor) CDR  minutes.  The  minutes  should  clearly  record  the  disposition 
of  all  critical  issues,  other  agreements  and  all  action  items,  including 
suspense  dates,  and  assign  specific  responsibility  to  the  Procuring 
Activity  and/or  the  contractor.  The  Procuring  Activity  provides  formal 
acknowledgement  to  the  contractor  of  the  accomplishment  of  the  CDR  after 
receipt  of  the  CDR  minutes,  but  should  not  formally  "approve”  the  design. 

4.5  TEST  READINESS  REVIEW  (TRR) 

The  Test  Readiness  Review  is  an  informal  review  and  is  not  required 
by  MIL-STD-1521A  (USAF).  The  TRR  is  a commonly  used  internal  review 
by  the  development  contractor  to  review  development  test  results  and 
evaluate  preparations  for  qualification  testing,  including  the  CPCI  con- 
figuration control  approach/procedures,  prior  to  commencing  qualification/ 
acceptance  testing.  Customer  attendance  is  optional,  but  as  a minimum 
the  procuring  activity  should  be  apprised  of  the  results  of  the  internal 
review.  Table  4-5  provides  a summary  of  the  TRR. 

Many  CPCI  development  contractors  establish  an  Internal  Review 
Board  (IRB),  at  die  outset  of  a CPCI  development,  composed  of  appropriate 
senior  personnel  not  otherwise  involved  in  the  project  activity.  This 
group  conducts  the  TRR,  dry  runs  the  formal  reviews  and  audits,  and 
schedules  other  internal  technical  reviews  at  significant  milestones 
(e.g.,  review  PQT  results)  intermediate  to  baselines,  at  the  discretion 
of  the  IRB  chairman.  A contractor  project  office  representative  may 
•erve  as  IRB  Secretary. 


Table  4-5.  Summary:  Teat  Readinese  Review  (TRR  );  Acquisition 
of  C PCI's  for  Airborne  Systems 


4.5.1  Leverage  Issues 

The  key  decision  at  the  TRR  for  a CPCI  is  whether  or  not  project 
preparations  to  commence  qualification/acceptance  testing  are  adequate. 
The  Internal  Review  Board  (XRB) 

a.  Ensures  the  adequacy,  traceability  and  completeness  of 
accomplished  development  testing  and  planned  qualification/ 
acceptance  testing  against  the  requirements  of  the  Part  I 
Development  Specification. 

b.  Evaluates  the  preparations  for  qualification/acceptance 
testing  (procedures,  tools/facilities,  personnel,  CPCI 
configuration  control  approach/procedures,  etc.). 

4.5.2  TRR  Post-Review  Action 


After  completion  of  the  TRR,  minutes  are  prepared,  signed  by  die 
IRB  chairman  and  distributed  to  project  personnel  and  top  management. 
The  minutes  should  clearly  record  all  action  items,  assignment  and 
suspense  dates,  and  ths  board's  recommendations. 

4.6  FUNCTIONAL  CONFIGURATION  AUDIT  (FCA) 

The  Functional  Configuration  Audit  (FCA)  verifies  the  CPCI's  actual 
(test)  performance  compliance  with  the  Part  I Development  Specification 
requirements.  Test  data  are  reviewed  to  verify  that  the  CPCI  met  all  of 
die  requirements  associated  with  its  Allocated  Baseline.  For  CPCI's 
developed  at  government  expense,  a satisfactory  FCA  is  a prerequisite  to 

acceptance.  The  FCA  for  a complex  CPCI  may  be  conducted  on  a 
progressive  basis,  when  so  specified  by  the  Procuring  Activity,  throughout 
the  CPCI  development  and  culminates  at  the  completion  of  qualification 
testing  of  the  CPCI  with  a review  of  all  discrepancies  at  the  final  FCA. 

For  CPCI's  that  can  be  validated  only  through  integrated  systems  testing, 
the  FCA  cannot  be  completed  until  such  testing  has  been  completed  and 
audited.  This  usually  implies  a separate  Formal  Qualification  Review 
(FQR)  (see  subsection  4.8).  Table  4-6  provides  a summary  of  the  FCA. 


Table  4-6.  Summary:  Functional  Configuration  Audit  (FCA);  Acquisition 
of  CPCI's  for  Airborne  Systems 


l 

I 

m 


£ 
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4.6.1  Leverage  Issues 

The  key  decision  associated  with  FCA  is  whether  or  not  the 
ensemble  of  CPC  I testing  satisfies  all  requirements  of  the  Part  1 
Development  Specification  (see  Table  4-6  for  details). 

4.6.2  FCA  Data  Packages 

The  contractor  provides  two  FCA  data  packages  to  the  customer: 

1.  data  items  to  be  delivered  20  days  or  some  negotiated 
lead  time  prior  to  FCA;  i.  e. , 

• a list  of  contractor  representatives,  including  the 
test  manager  or  equivalent;  and 

• identification  of  the  CPC  I to  be  audited,  including 

• nomenclature  (name  or  descriptive  title  of 
the  CPCI, 

• specification  identification  number  and  the  CDRL 
identifier  of  the  document,  and 

• CPCI  identifier; 

2.  documentation  and  data  to  be  provided  and  made  available 

to  the  customer  at  the  FCA  (Table  4-6,  "Items  to  be  Reviewed") 

4.6.3  FCA  Post-Audit  Action 

After  completion  of  the  FCA,  the  contractor  is  responsible  for 
publishing  and  distributing  the  official  (co-signed  by  customer  and  con- 
tractor) FCA  minutes.  The  minutes  should  clearly  record  all  results 
and  findings,  including  a discussion  of  all  deficiencies.  The  Procuring 
Activity  provides  formal  acknowledgement  to  the  contractor  of  the 
accomplishment  of  the  FCA  after  receipt  of  the  FCA  minutes. 

4.7  PHYSICAL  C ON F DURATION  AUDIT  (PCA) 

The  PCA  is  a formal  examination  of  the  coded  version  of  a CPCI 
against  its  technical  documentation  and  of  the  configuration  management 
records  pertinent  to  the  CPCI  in  order  to  establish  the  Product  Baseline. 


The  PCA  cannot  be  conducted  unless  the  customer  has  the  final  draft  of 
the  Part  II  Product  Specification  (nominally  at  least  30  days  prior  to  PCA). 
After  successful  completion  of  the  PCA  and  formal  customer  approval 
of  the  Product  Specification,  all  subsequent  changes  are  processed  by 
ECP.  Table  4-7  provides  a summary  of  the  PCA. 

4.7.1  Leverage  Issues 

The  key  decision  at  the  PCA  is  whether  to  approve  the  Part  II 
Product  Specification  establishing  the  CPCI  product  baseline,  thus 
formally  "accepting"  the  CPCI.  The  procuring  activity  should 

a.  Conduct  a detailed  audit  of  the  Part  II  Product  Specification, 
including  its  flow  charts,  listings,  and  design  narrative. 

Also  review,  for  format  and  completeness,  the  Operator's 
(User's)  Manual, Computer  Programming  Manual  and  any 
other  manuals  and  handbooks  specified  in  the  contract; 
these  manuals  and  handbooks  are  reviewed  and  analyzed 

for  final  approval  after  integrated  systems  test  has  verified 
that  procedures  are  accurate. 

b.  Review  configuration  management  status  accounting  records 
related  to  the  CPCI  to  ensure  that  all  approved  changes  are 
incorporated  and  that  unapproved  changes  are  not  incorporated, 
but  properly  logged. 

c.  Evaluate  all  CPCI  configuration  differences  between  FCA 
and  PCA  to  ensure  CPCI  functional  characteristics  are 
not  degraded. 

Satisfactory  completion  of  the  PCA  and  formal  approval  (DD  Form 
250  or  equivalent)  of  the  CPCI  Part  II  Product  Specification  establishes 
the  CPCI  Product  Baseline. 

4.7.2  PCA  Data  Packages 

The  contractor  provides  three  PCA  data  packages  to  the  customer: 

1.  Final  draft  Part  II  Product  Specification  for  the  CPCI  to 
by  PCA'd  nominally  at  least  30  days  prior  to  the  PCA; 

2.  Data  items  to  be  delivered  20  days  or  some  negotiated 
lead  time  prior  to  PCA: 

• PCA  date  and  location 

• agenda 


. 


Table  4-7.  Summary:  Physical  Configuration 
Audit  (PCA);  Acquisition  of  CPCl'a 
For  Airborne  Systems 


Items  to  Be  Reviewed 


Formal  Documentation 


Other  Technical /Management  Data 
(Contractor  Presentations, 
Internal  Report*,  etc.) 


Conducted  between 
FCA  and  FOR 
(when  separate 
FOR  is  required) 
when  all  required 
audit  data  is  avail- 
able and  (nominally) 
at  least  30  days  after 
submittal  of  the  draft 
Final  Part  II  Product 
Specification  to  the 
Procuring  Activity. 


• Formal  examination  of  coded  ("as  build")  version 
of  the  CPCI  against  its  technical  documentation  and 
of  the  configuration  management  records  pertinent 
to  the  CPCI  to  establish  the  Product  Baseline. 

e Review  Part  II  Product  Specification  for  format 
and  completeness. 

• Review  FCA  minutqs.  for  recorded  discrepancies 
that  require  action. 

e Review  Computer  Program  Component  (CPC) 
description  and  flow  charts. 

e Review  CPC  Interface  requirements, 
e Review  data  base  characteristics,  storage 
allocation  charts,  and  timing  and  sequencing 
characteristics. 

• Review  flow  charts  for  proper  entries,  symbols, 
and  label  tags. 

e Review  acceptance  test  procedures /results  for 
compliance  with  Part  II  Product  Specification. 

e Compare  top  level  CPCI  flow  charts  with  CPC 
flow  charts. 

e Compare  detailed  CPC  flow  charts  with  coded 
program  (listings)  for  accuracy  and  completeness. 
Comparison  may  be  performed  using  a sampling 
rather  than  exhaustive  techniques.  The  sampling 
rate  should  be  adjusted  based  upon  observed 
compatibility. 

e Check  Computer  Programming  Manual,  Operator's 
(User's)  Manual,  and  Version  Description  Docu- 
ment for  format,  completeness  and  conformance 
with  applicable  data  items.  (Formal  verification/ 
acceptance  of  these  handbooks  /manuals  should  be 
withheld  until  system  testing  to  ensure  that  the 
procedural  contents  are  correct). 

e Cross-check  current  (code)  listing  with  the  listing 
in  the  Part  II  Product  Specification.  The  listing 
roty  be  cor ss -checked  using  a sampling  rather 
than  an  exhaustive  technique.  The  sampling  rate 
should  be  adjusted  according  to  the  observed 
compatibility. 

e Examine  actual  CPCI  (card  decks,  tapes,  etc.) 
for  conformance  with  Section  5 of  the  Part  II 
Product  Specification 

• Evaluate  all  CPCI  configuration  differences  between 
FCA  and  PCA  versions  to  ensure  CPCI  functional 
characteristics  are  not  degraded.  (PCA  Minutes). 

• Audit  the  contractor's  engineering  release  and  cnange 
control  system  to  ensure  that  it  is  adequate  to  properly 
control  the  processing  and  formal  release  of 
engineering  changes.  As  a minimum,  assure  the 
capability  to  accomplish: 

e Identification  of  changes  and  retaining  records  of 
superseded  configuration  formally  accepted  by 
the  Processing  Activity. 

e Identification  and  accountability  of  all  Class  I and  II 
engineering  changes  released  for  incorporation. 
These  changes  should  be  completely  released  and 
incorporated  prior  to  formal  acceptance  of  the  CPCI. 

e Determination  of  die  configuration  release  for  each 
CPCI  at  the  time  for  formal  acceptance. 

e Processing  and  release  of  engineering  data  through 
a central  authority  to  ensure  coordinated  action 
and  preclude  unilateral  release  of  data. 

• Satisfactory  completion  of  the  PCA  and  formal  approval 
by  the  Procuring  Activity  of  the  CPCI  Part  II  Product 
Specification  establlshs  the  CPCI  Product  Baseline. 


• Authenticated  CPCI 
Part  I Development 
Specification. 

• Draft  Final  CPCI  Part 
II  Product  Specification. 

• CPCI  Test  Plans,  Test 
Procedures,  and  Test 
Reports. 

• Draft  Final  Operator's 
(User's)  Manual. 

• Draft  Final  Computer 
Programming  Manual. 

• Version  Description 
Document  (VDD). 

• FCA  Minutes. 

• Prepared  DD  Form  250 
or  equivalent. 


List  of  all  deviations /waivers  against  the 
CPCI,  either  approved,  or  outstanding 
awaiting  approval  by  the  Procuring  Activity. 

List  delineating  both  approved  and  outstanding 
changes  (ECP's)  against  the  CPCI. 

List  of  all  required  changes  not  yet  completed. 
List  of  ail  changes  actually  made  during  test. 

CPCI  configuration  management  status 
accounting  records. 

CPCI  master  tape  and  current  listings  and 
flow  charts. 

Description  of  contractor's  engineering 
release  and  configuration  control  system. 


• Accepted  CPCI's  are  delivered  1 
contract  requirements. 


i accordance  with 


Tfc.  two  ilmMi  that  dl«ln,ulah  "formal  baaeltn.,  from  a "lnt.rm.diat.  mil. .ton..-  ara  that  (1)  formal  baaalinaa  involve  formal  coatomor 
approval  of  configuration  documentation  (.pacification),  and  (2)  approved  apocificationa  are  pul  under  formal  confipu ration  control. 


I Formal  Baaollne*Eatabllahodt|  intermediate  Mile. 
Product  Baaetiae  I N/A 
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• list  of  contractor  representatives,  including  the 
test  manager  or  equivalent 

9 identification  of  CPC1  to  be  audited/accepted: 

# nomenclature  (name  or  descriptive  title  of  the  CPCI) 

# specification  identification  number  and  the  CDRL 
identifier  of  the  document 

# CPCI  identifier 

# CPC,  module  and/or  routine  identifiers 

• list  of  all  deviations  /waivers  against  the  CPCI,  either 
requested  or  Procuring  Activity  approved;  and 

3.  documentation  and  data  to  be  provided  and  made  available 
at  the  PC  A (Table  4-7,  "Items  to  be  Reviewed"). 

4.7.3  PCA  Post-Audit  Action 

After  completion  of  the  PCA,  the  contractor  is  responsible  for 
publishing  and  distributing  the  official  (co-signed  by  customer  and  con- 
tractor) PCA  minutes.  The  minutes  should  clearly  record  all  results 
and  findings,  including  a tabulation/discussion  of  all  deficiencies;  action 
item  assignments  should  address  all  deficiencies  (e.g.  , make  corrections, 
prepare /execute  waivers,  etc.)-  The  Procuring  Activity  provides  formal 
acknowledgement  to  the  contractor  that  PCA  took  place  after  receipt  of 
the  PCA  minutes. 

Procuring  Activity  acceptance  or  rejection  of  the  CPCI  and  the 
CPCI  Part  II  Product  Specification  must  be  furnished  to  the  contractor 
in  writing  by  the  responsible  contract  management  agency  or  other 
designated  agency  after  completion  of  PCA. 

4.8  FORMAL  QUALIFICATION  REVIEW  (FQR) 

When  feasible,  the  FQR  is  combined  with  the  FCA.  For  situations 
in  which  the  CPCI  Part  I Development  Specification  requirements  cannot 
totally  be  verified  by  the  testing  accomplished  at  FCA  (e.g. , CPCI 
qualification  dependent  on  integrated  system  testing),  a separate  FQR 
should  be  conducted  post-PCA,  when  the  necessary  tests  have  been 
satisfactorily  completed,  to  enable  CPCI  certification.  Table  4-8 
provides  a summary  of  the  FQR. 


Table  4-8..  Summary:  Formal  Qualification  Review  (FQR);  Acquisition  of  CPCI's  for  Airborne  Systems 


Formal  Bueline*  Eittbliihtd:  Intermediate  Milestone: 
N/A  Qualified  CPCI 


4.8.1  Leverage  Issues 


The  key  decision  associated  with  the  FQR  Cor  a CPCI,  assuming 
a separate  FQR  is  required,  is  whether  or  not  the  combination  of  the 
qualification/acceptance  testing  audited  at  FCA  and  the  post-FCA  testing 
satisfies  all  requirements  of  the  Part  I Development  Specification.  The 
Procuring  Activity  should 

a.  review  FCA  minutes  to  ensure  that  all  findings  have  been 
incorporated  and  completed  (the  FQR  is  considered  an 
extension  of  the  FCA);  and 

b.  review  and  evaluate  additional  qualification/acceptance 
testing  data  acquired  post-FCA.  (This  additional  data 
and  the  FCA  findings  should  verify  tha'.  the  CPCI  satisfies 
all  of  the  requirements  in  the  CPCI  Part  I Development 
Specification). 

4.8.2  FQR  Data  Packages 

These  are  the  same  as  the  data  packages  for  FCA,  plus: 

e additional  test  results,  and 

s FCA  minutes. 

4.8.3  FQR  Post-Audit  Action 

The  required  poat-audit  activities  are  identical  to  those  required 
for  the  FCA. 


i 
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APPENDIX 

REVIEWS  AND  AUDITS  GUIDEBOOK 
BIBLIOGRAPHY  OF  GOVERNMENT  DOCUMENTS 


Designator 

Version 

Date 

Title 

DODD  5000.  19 

6/02/71 

Policies  for  the  Management  and 
Control  of  DoD  Information 
Requirements 

DODD  5000.  29 

4/26/76 

Management  of  Computer  Resources 
in  Major  Defense  Programs 

DODD  5010.  28 

10/02/72 

Department  of  Defense  Management 
Review  and  Improvement  Program 

DODI  4105.64 

8/05/70 

Technical  Representation  at 
Contractor  Facilities 

AFR  173-1 

6/29/73 

Management  of  the  Cost  Analysis 
Program 

AFR  174-2 

5/17/68 

Follow-Up  on  Internal  Reports  of 
Audit  (AFSC  Supplement  11/27/72 
and  ESC  Supplement  6/15/72) 

AFR  175-4 

11/14/72 

Auaiting  in  the  Air  Force 

AFR  800-5 

7/27/73 

Selected  Acquisition  Reports  (SARs) 

AFR  800-14 

Volume  I 

5/10/74 

Management  of  Computer  Resources 
in  Systems  (AFSC  Supplement 
9/25/74) 

AFR  800-14 

Volume  II 

9/26/75 

Acquisition  and  Support  of  Computer 
Resources  in  Systems 

AFP  70-14 

3/01/74 

PIECOST  (Probability  of  Incurring 
Estimated  Cost) 

AFM  175-118 

5/17/74 

Air  Force  Audit /Management  System 

AFSCR  70-12 

11/29/74 

AFSC  Procurement  Summary  Report 

AFSCR  800-1 

4/24/74 

Command  Review  of  Systems 
Acquisition  Programs  and  Test 
Resources 

AFSCR  800-18 

9/20/74 

Joint  Operational  and  Technical 
Review  (JOTR) 

Designator 

Version 

Date 

Title 

AFSCP  800-3 

4/09/76 

A Guide  for  Program  Management 

ESDR  27-4 

5/10/73 

Documentation  for  System 

Program  Reviews 

MIL-S-52779(AD) 

4/05/74 

Software  Quality  Assurance 
Program  Requirements 

MIL -STD-480 

10/31/68 

Configuration  Control-Engineering 
changes.  Deviations  and  Waivers 

MIL -STD-481  A 

Configuration  Control  - Engineer- 
ing Changes,  Deviations,  and 
Waivers  (Short  Form) 

MIL-STD-483  (USAF) 

6/01/71 

Configuration  Management 

Practices  for  Systems,  Equipment, 
Munitions,  and  Computer  Programs 

MIL-STD-490 

10/30/68 

Specification  Practices 

MIL-STD-499A  (USAF) 

5/01/74 

Engineering  Management 

MIL-STD-1521A  (USAF) 

6/01/76 

Technical  Reviews  and  Audits  for 
Systems,  Equipment,  and 

Computer  Programs 

MIL-STD- 1602 

6/08/73 

Requirements  for  Progress  Reports 

for  R&D  Equipment 


